Structured Prediction Crosswalk Generation

ABSTRACT

A method includes receiving image data associated with an image of a roadway including a crosswalk, generating a plurality of different characteristics of the image based on the image data, determining a position of the crosswalk on the roadway based on the plurality of different characteristics, the position including a first boundary and a second boundary of the crosswalk in the roadway, and providing map data associated with a map of the roadway, the map data including the position of the crosswalk on the roadway in the map. The plurality of different characteristics include a classification of one or more elements of the image, a segmentation of the one or more elements of the image, and one or more angles of the one or more elements of the image with respect to a line in the roadway.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/642,835, filed Mar. 14, 2018, the entire disclosure of which ishereby incorporated by reference in its entirety.

BACKGROUND

An autonomous vehicle (e.g., a driverless car, a driverless automobile,a self-driving car, a robotic car, etc.) is a vehicle that is capable ofsensing an environment of the vehicle and traveling (e.g., navigating,moving, etc.) in the environment without human input. An autonomousvehicle uses a variety of techniques to detect the environment of theautonomous vehicle, such as radar, laser light, Global PositioningSystem (GPS), odometry, and/or computer vision. In some instances, anautonomous vehicle uses a control system to interpret informationreceived from one or more sensors, to identify a route for traveling, toidentify an object in a route, and to identify relevant roadway markingsand signs associated with a route.

SUMMARY

Accordingly, provided are improved systems, devices, products,apparatus, and/or methods for automatically drawing, predicting, and/orproviding a crosswalk.

According to some non-limiting embodiments or aspects, provided is acomputer-implemented method comprising: receiving, with a computingsystem comprising one or more processors, image data associated with animage of a roadway including a crosswalk; generating, with the computingsystem, a plurality of different characteristics of the image based onthe image data, wherein the plurality of different characteristicsinclude a classification of one or more elements of the image, asegmentation of the one or more elements of the image, and one or moreangles of the one or more elements of the image with respect to a linein the roadway; determining, with the computing system, a position ofthe crosswalk on the roadway based on the plurality of differentcharacteristics, wherein the position includes a first boundary and asecond boundary of the crosswalk in the roadway; and providing, with thecomputing system, map data associated with a map of the roadway, whereinthe map data includes the position of the crosswalk on the roadway inthe map.

In some non-limiting embodiments or aspects, determining the position ofthe crosswalk on the roadway based on the plurality of differentcharacteristics further comprises: determining, based on the one or moreangles and the image data, a first likelihood of the one or moreelements being on the first boundary or the second boundary;determining, based on the one or more angles and the image data, asecond likelihood of the one or more elements being between the firstboundary and the second boundary; and determining, based on the firstlikelihood and the second likelihood, whether the position of thecrosswalk on the roadway includes the one or more elements.

In some non-limiting embodiments or aspects, the position of thecrosswalk on the roadway is determined according to the followingfunction:

${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$

and ϕ_(seg) is a segmented output feature map including the segmentationof the one or more elements in the image, ϕ_(dt) is a classificationoutput feature map including the classification of the one or moreelements in the image, χ₁ and χ₂ are points on the line l in theroadway, β is the one or more angles of the one or more elements of theimage with respect to the line l, and λ_(I) is a weight that balancesϕ_(seg) and ϕ_(dt).

In some non-limiting embodiments or aspects, the segmentation of the oneor more elements of the image includes a segmentation image including aplurality of elements inside the first boundary and the second boundaryand a plurality of elements outside the first boundary and secondboundary.

In some non-limiting embodiments or aspects, the classification of theone or more elements of the image includes a distance map including aplurality of elements associated with at least one of the first boundaryand the second boundary, the distance map including at least onedistance to the at least one of the first boundary and the secondboundary for at least one element of the plurality of elements within athreshold distance of the at least one of the first boundary and thesecond boundary.

In some non-limiting embodiments or aspects, the line includes at leasta portion of a centerline of the roadway.

In some non-limiting embodiments or aspects the one or more angles ofthe one or more elements comprise a dilated representation of a boundaryangle including an x-value and an y-value with respect to the line.

In some non-limiting embodiments or aspects, the method furthercomprises training, with the computing system, a convolutional neuralnetwork by modifying parameters to optimize a loss function of theconvolutional neural network that depends on ground truth labels and oneor more predictions of the plurality of different characteristics,wherein the loss function comprises a sum of a segmentation loss, analignment loss, and a boundary loss; processing, with the computingsystem, the image data associated with the image of the roadwayincluding the crosswalk using the trained convolutional neural network,to generate a plurality of images associated with the plurality ofdifferent characteristics of the image.

According to some non-limiting embodiments or aspects, provided is acomputing system comprising: one or more processors programmed and/orconfigured to: receive image data associated with an image of a roadwayincluding a crosswalk; generate a plurality of different characteristicsof the image based on the image data, wherein the plurality of differentcharacteristics include a classification of one or more elements of theimage, a segmentation of the one or more elements of the image, and oneor more angles of the one or more elements of the image with respect toa line in the roadway; determine a position of the crosswalk on theroadway based on the plurality of different characteristics, wherein theposition includes a first boundary and a second boundary of thecrosswalk in the roadway; and provide map data associated with a map ofthe roadway, wherein the map data includes the position of the crosswalkon the roadway in the map.

In some non-limiting embodiments or aspects, the one or more processorsare further programmed or configured to determine the position of thecrosswalk on the roadway based on the plurality of differentcharacteristics by: determining, based on the one or more angles and theimage data, a first likelihood of the one or more elements being on thefirst boundary or the second boundary; determining, based on the one ormore angles and the image data, a second likelihood of the one or moreelements being between the first boundary and the second boundary; anddetermining, based on the first likelihood and the second likelihood,whether the position of the crosswalk on the roadway includes the one ormore elements.

In some non-limiting embodiments or aspects, the position of thecrosswalk on the roadway is determined according to the followingfunction:

${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$

wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁ and χ₂ are points on the line l inthe roadway, β is the one or more angles of the one or more elements ofthe image with respect to the line l, and λ_(I) is a weight thatbalances ϕ_(seg) and ϕ_(dt).

In some non-limiting embodiments or aspects, the segmentation of the oneor more elements of the image includes a segmentation image including aplurality of elements inside the first boundary and the second boundaryand a plurality of elements outside the first boundary and secondboundary.

In some non-limiting embodiments or aspects, the classification of theone or more elements of the image includes a distance map including aplurality of elements associated with at least one of the first boundaryand the second boundary, the distance map including at least onedistance to the at least one of the first boundary and the secondboundary for at least one element of the plurality of elements within athreshold distance of the at least one of the first boundary and thesecond boundary.

In some non-limiting embodiments or aspects, the line includes at leasta portion of a centerline of the roadway.

In some non-limiting embodiments or aspects, the one or more angles ofthe one or more elements comprise a dilated representation of a boundaryangle including an x value and a y value with respect to the line.

In some non-limiting embodiments or aspects, the one or more processorsare further programmed and/or configured to: train a convolutionalneural network by modifying parameters to optimize a loss function ofthe convolutional neural network that depends on ground truth labels andone or more predictions of the plurality of different characteristics,wherein the loss function comprises a sum of a segmentation loss, analignment loss, and a boundary loss; and process the image dataassociated with the image of the roadway including the crosswalk usingthe trained convolutional neural network, to generate a plurality ofimages associated with the plurality of different characteristics of theimage.

According to some non-limiting embodiments or aspects, provided is acomputer program product comprising at least one non-transitorycomputer-readable medium including program instructions that, whenexecuted by at least one processor, cause the at least one processor to:receive image data associated with an image of a roadway including acrosswalk; generate a plurality of different characteristics of theimage based on the image data, wherein the plurality of differentcharacteristics include a classification of one or more elements of theimage, a segmentation of the one or more elements of the image, and oneor more angles of the one or more elements of the image with respect toa line in the roadway; determine a position of the crosswalk on theroadway based on the plurality of different characteristics, wherein theposition includes a first boundary and a second boundary of thecrosswalk in the roadway; and provide map data associated with a map ofthe roadway, wherein the map data includes the position of the crosswalkon the roadway in the map.

In some non-limiting embodiments or aspects, the instructions cause theat least one processor to determine the position of the crosswalk on theroadway based on the plurality of different characteristics by:determining, based on the one or more angles and the image data, a firstlikelihood of the one or more elements being on the first boundary orthe second boundary; determining, based on the one or more angles andthe image data, a second likelihood of the one or more elements beingbetween the first boundary and the second boundary; and determining,based on the first likelihood and the second likelihood, whether theposition of the crosswalk on the roadway includes the one or moreelements

In some non-limiting embodiments or aspects, the position of thecrosswalk on the roadway is determined according to the followingfunction:

${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$

wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁and χ₂ are points on the line l inthe roadway, β is the one or more angles of the one or more elements ofthe image with respect to the line l, and λ_(I) is a weight thatbalances ϕ_(seg) and ϕ_(dt).

In some non-limiting embodiments or aspects the line includes at least aportion of a centerline of the roadway.

According to some non-limiting embodiments or aspects, provided is anautonomous vehicle comprising a vehicle computing system that comprisesone or more processors, wherein the vehicle computing system isconfigured to: receive image data associated with an image of a roadwayincluding a crosswalk; generate a plurality of different characteristicsof the image based on the image data, wherein the plurality of differentcharacteristics include a classification of one or more elements of theimage, a segmentation of the one or more elements of the image, and oneor more angles of the one or more elements of the image with respect toa line in the roadway; determine a position of the crosswalk on theroadway based on the plurality of different characteristics, wherein theposition includes a first boundary and a second boundary of thecrosswalk in the roadway; and provide map data associated with a map ofthe roadway, wherein the map data includes the position of the crosswalkon the roadway in the map.

Further non-limiting embodiments or aspects are set forth in thefollowing numbered clauses:

Clause 1. A computer-implemented method, comprising: receiving, with acomputing system comprising one or more processors, image dataassociated with an image of a roadway including a crosswalk; generating,with the computing system, a plurality of different characteristics ofthe image based on the image data, wherein the plurality of differentcharacteristics include a classification of one or more elements of theimage, a segmentation of the one or more elements of the image, and oneor more angles of the one or more elements of the image with respect toa line in the roadway; determining, with the computing system, aposition of the crosswalk on the roadway based on the plurality ofdifferent characteristics, wherein the position includes a firstboundary and a second boundary of the crosswalk in the roadway; andproviding, with the computing system, map data associated with a map ofthe roadway, wherein the map data includes the position of the crosswalkon the roadway in the map.

Clause 2: The computer-implemented method of clause 1, whereindetermining the position of the crosswalk on the roadway based on theplurality of different characteristics further comprises: determining,based on the one or more angles and the image data, a first likelihoodof the one or more elements being on the first boundary or the secondboundary; determining, based on the one or more angles and the imagedata, a second likelihood of the one or more elements being between thefirst boundary and the second boundary; and determining, based on thefirst likelihood and the second likelihood, whether the position of thecrosswalk on the roadway includes the one or more elements.

Clause 3: The computer-implemented method of clauses 1 and 2, whereinthe position of the crosswalk on the roadway is determined according tothe following function:

${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$

wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁ and χ₂ are points on the line l inthe roadway, β is the one or more angles of the one or more elements ofthe image with respect to the line l, and λ_(I) is a weight thatbalances ϕ_(seg) and ϕ_(dt).

Clause 4: The computer-implemented method of clauses 1-3, wherein thesegmentation of the one or more elements of the image includes asegmentation image including a plurality of elements inside the firstboundary and the second boundary and a plurality of elements outside thefirst boundary and second boundary.

Clause 5: The computer-implemented method of clauses 1-4, wherein theclassification of the one or more elements of the image includes adistance map including a plurality of elements associated with at leastone of the first boundary and the second boundary, the distance mapincluding at least one distance to the at least one of the firstboundary and the second boundary for at least one element of theplurality of elements within a threshold distance of the at least one ofthe first boundary and the second boundary.

Clause 6: The computer-implemented method of clauses 1-5, wherein theline includes at least a portion of a centerline of the roadway.

Clause 7: The computer-implemented method of clauses 1-6, wherein theone or more angles of the one or more elements comprise a dilatedrepresentation of a boundary angle including an x-value and an y-valuewith respect to the line.

Clause 8: The computer-implemented method of clauses 1-7, furthercomprising: training, with the computing system, a convolutional neuralnetwork by modifying parameters to optimize a loss function of theconvolutional neural network that depends on ground truth labels and oneor more predictions of the plurality of different characteristics,wherein the loss function comprises a sum of a segmentation loss, analignment loss, and a boundary loss; processing, with the computingsystem, the image data associated with the image of the roadwayincluding the crosswalk using the trained convolutional neural network,to generate a plurality of images associated with the plurality ofdifferent characteristics of the image.

Clause 9: A computing system, comprising: one or more processorsprogrammed and/or configured to: receive image data associated with animage of a roadway including a crosswalk; generate a plurality ofdifferent characteristics of the image based on the image data, whereinthe plurality of different characteristics include a classification ofone or more elements of the image, a segmentation of the one or moreelements of the image, and one or more angles of the one or moreelements of the image with respect to a line in the roadway; determine aposition of the crosswalk on the roadway based on the plurality ofdifferent characteristics, wherein the position includes a firstboundary and a second boundary of the crosswalk in the roadway; andprovide map data associated with a map of the roadway, wherein the mapdata includes the position of the crosswalk on the roadway in the map.

Clause 10: The computing system of clause 9, wherein the one or moreprocessors are further programmed or configured to determine theposition of the crosswalk on the roadway based on the plurality ofdifferent characteristics by: determining, based on the one or moreangles and the image data, a first likelihood of the one or moreelements being on the first boundary or the second boundary;determining, based on the one or more angles and the image data, asecond likelihood of the one or more elements being between the firstboundary and the second boundary; and determining, based on the firstlikelihood and the second likelihood, whether the position of thecrosswalk on the roadway includes the one or more elements.

Clause 11: The computing system of clauses 9 and 10, wherein theposition of the crosswalk on the roadway is determined according to thefollowing function:

${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$

wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁ and χ₂ are points on the line l inthe roadway, βis the one or more angles of the one or more elements ofthe image with respect to the line l, and λ_(I) is a weight thatbalances ϕ_(seg) and ϕ_(dt).

Clause 12: The computing system of clauses 9-11, wherein thesegmentation of the one or more elements of the image includes asegmentation image including a plurality of elements inside the firstboundary and the second boundary and a plurality of elements outside thefirst boundary and second boundary.

Clause 13: The computing system of clauses 9-12, wherein theclassification of the one or more elements of the image includes adistance map including a plurality of elements associated with at leastone of the first boundary and the second boundary, the distance mapincluding at least one distance to the at least one of the firstboundary and the second boundary for at least one element of theplurality of elements within a threshold distance of the at least one ofthe first boundary and the second boundary.

Clause 14: The computing system of clauses 9-13, wherein the lineincludes at least a portion of a centerline of the roadway.

Clause 15: The computing system of clauses 9-14, wherein the one or moreangles of the one or more elements comprise a dilated representation ofa boundary angle including an x value and a y value with respect to theline.

Clause 16: The computing system of clauses 9-15, wherein the one or moreprocessors are further programmed and/or configured to: train aconvolutional neural network by modifying parameters to optimize a lossfunction of the convolutional neural network that depends on groundtruth labels and one or more predictions of the plurality of differentcharacteristics, wherein the loss function comprises a sum of asegmentation loss, an alignment loss, and a boundary loss; and processthe image data associated with the image of the roadway including thecrosswalk using the trained convolutional neural network, to generate aplurality of images associated with the plurality of differentcharacteristics of the image.

Clause 17: A computer program product comprising at least onenon-transitory computer-readable medium including program instructionsthat, when executed by at least one processor, cause the at least oneprocessor to: receive image data associated with an image of a roadwayincluding a crosswalk; generate a plurality of different characteristicsof the image based on the image data, wherein the plurality of differentcharacteristics include a classification of one or more elements of theimage, a segmentation of the one or more elements of the image, and oneor more angles of the one or more elements of the image with respect toa line in the roadway; determine a position of the crosswalk on theroadway based on the plurality of different characteristics, wherein theposition includes a first boundary and a second boundary of thecrosswalk in the roadway; and provide map data associated with a map ofthe roadway, wherein the map data includes the position of the crosswalkon the roadway in the map.

Clause 18: The computer program product of clause 17, wherein theinstructions cause the at least one processor to determine the positionof the crosswalk on the roadway based on the plurality of differentcharacteristics by: determining, based on the one or more angles and theimage data, a first likelihood of the one or more elements being on thefirst boundary or the second boundary; determining, based on the one ormore angles and the image data, a second likelihood of the one or moreelements being between the first boundary and the second boundary; anddetermining, based on the first likelihood and the second likelihood,whether the position of the crosswalk on the roadway includes the one ormore elements.

Clause 19: The computer program product of clause 17 and 18, wherein theposition of the crosswalk on the roadway is determined according to thefollowing function:

${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$

wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁ and χ₂ are points on the line l inthe roadway, β is the one or more angles of the one or more elements ofthe image with respect to the line l, and λ_(I) is a weight thatbalances ϕ_(seg) and ϕ_(dt).

Clause 20: The computer program product of clause 17-19, wherein theline includes at least a portion of a centerline of the roadway.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of non-limiting embodiments or aspects of anenvironment in which systems, devices, products, apparatuses, and/ormethods, described herein, can be implemented;

FIG. 2 is a diagram of non-limiting embodiments or aspects of a systemfor controlling an autonomous vehicle shown in FIG. 1;

FIG. 3 is a diagram of non-limiting embodiments or aspects of componentsof one or more devices and/or one or more systems of FIGS. 1 and 2;

FIG. 4 is a flowchart of non-limiting embodiments or aspects of aprocess for crosswalk generation;

FIG. 5 is a diagram of non-limiting embodiments or aspects of acrosswalk generation model;

FIG. 6 is a diagram of non-limiting embodiments or aspects of aconvolutional neural network for map generation;

FIG. 7 includes images of non-limiting embodiments or aspects ofmultiple pass RGB and LIDAR (“RL”) crosswalk inputs, networkimplementation outputs, predicted crosswalk polygons, and ground truthcrosswalk polygons.

FIG. 8 includes images of non-limiting embodiments or aspects of singlepass RL crosswalk inputs, network implementation outputs, predictedcrosswalk polygons, and ground truth crosswalk polygons;

FIG. 9 includes graphs of non-limiting embodiments or aspects ofintersection over union (“IOU”) associated with multiple pass RLimplementations;

FIG. 10 includes graphs of non-limiting embodiments or aspects of angleprediction associated with single pass RL implementations;

FIGS. 11A-11C are graphs of non-limiting embodiments or aspects of anumber of roads associated with implementations;

FIG. 12 includes images of non-limiting embodiments or aspects offailure modes in comparisons of crosswalk inputs, network implementationoutputs, predicted crosswalk polygons, and ground truth crosswalkpolygons;

FIG. 13 includes images of non-limiting embodiments or aspects ofbalancing between segmentation and distance transform weights and poorimage/paint quality in comparisons of crosswalk inputs, networkimplementation outputs, predicted crosswalk polygons, and ground truthcrosswalk polygons;

FIG. 14 includes images of non-limiting embodiments or aspects ofcomparisons of crosswalk inputs, network implementation outputs,predicted crosswalk polygons, and ground truth crosswalk polygons ofonline model crosswalks having failure modes; and

FIG. 15 is a diagram of non-limiting embodiments or aspects of acrosswalk generation model.

DETAILED DESCRIPTION

It is to be understood that the present disclosure may assume variousalternative variations and step sequences, except where expresslyspecified to the contrary. It is also to be understood that the specificdevices and processes illustrated in the attached drawings, anddescribed in the following specification, are simply exemplary andnon-limiting embodiments or aspects. Hence, specific dimensions andother physical characteristics related to the embodiments or aspectsdisclosed herein are not to be considered as limiting.

For purposes of the description hereinafter, the terms “end,” “upper,”“lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,”“lateral,” “longitudinal,” and derivatives thereof shall relate toembodiments or aspects as they are oriented in the drawing figures.However, it is to be understood that embodiments or aspects may assumevarious alternative variations and step sequences, except whereexpressly specified to the contrary. It is also to be understood thatthe specific devices and processes illustrated in the attached drawings,and described in the following specification, are simply non-limitingexemplary embodiments or aspects. Hence, specific dimensions and otherphysical characteristics related to the embodiments or aspects of theembodiments or aspects disclosed herein are not to be considered aslimiting unless otherwise indicated.

No aspect, component, element, structure, act, step, function,instruction, and/or the like used herein should be construed as criticalor essential unless explicitly described as such. Also, as used herein,the articles “a” and “an” are intended to include one or more items, andmay be used interchangeably with “one or more” and “at least one.”Furthermore, as used herein, the term “set” is intended to include oneor more items (e.g., related items, unrelated items, a combination ofrelated and unrelated items, etc.) and may be used interchangeably with“one or more” or “at least one.” Where only one item is intended, theterm “one” or similar language is used. Also, as used herein, the terms“has,” “have,” “having,” or the like are intended to be open-endedterms. Further, the phrase “based on” is intended to mean “based atleast partially on” unless explicitly stated otherwise.

As used herein, the terms “communication” and “communicate” may refer tothe reception, receipt, transmission, transfer, provision, and/or thelike of information (e.g., data, signals, messages, instructions,commands, and/or the like). For one unit (e.g., a device, a system, acomponent of a device or system, combinations thereof, and/or the like)to be in communication with another unit means that the one unit is ableto directly or indirectly receive information from and/or transmitinformation to the other unit. This may refer to a direct or indirectconnection that is wired and/or wireless in nature. Additionally, twounits may be in communication with each other even though theinformation transmitted may be modified, processed, relayed, and/orrouted between the first and second unit. For example, a first unit maybe in communication with a second unit even though the first unitpassively receives information and does not actively transmitinformation to the second unit. As another example, a first unit may bein communication with a second unit if at least one intermediary unit(e.g., a third unit located between the first unit and the second unit)processes information received from the first unit and communicates theprocessed information to the second unit. In some non-limitingembodiments or aspects, a message may refer to a network packet (e.g., adata packet and/or the like) that includes data. It will be appreciatedthat numerous other arrangements are possible.

As used herein, the term “computing device” may refer to one or moreelectronic devices that are configured to directly or indirectlycommunicate with or over one or more networks. A computing device may bea mobile or portable computing device, a desktop computer, a server,and/or the like. Furthermore, the term “computer” may refer to anycomputing device that includes the necessary components to receive,process, and output data, and normally includes a display, a processor,a memory, an input device, and a network interface. A “computing system”may include one or more computing devices or computers. An “application”or “application program interface” (API) refers to computer code orother data sorted on a computer-readable medium that may be executed bya processor to facilitate the interaction between software components,such as a client-side front-end and/or server-side back-end forreceiving data from the client. An “interface” refers to a generateddisplay, such as one or more graphical user interfaces (GUIs) with whicha user may interact, either directly or indirectly (e.g., through akeyboard, mouse, touchscreen, etc.). Further, multiple computers, e.g.,servers, or other computerized devices, such as an autonomous vehicleincluding a vehicle computing system, directly or indirectlycommunicating in the network environment may constitute a “system” or a“computing system”.

It will be apparent that systems and/or methods, described herein, canbe implemented in different forms of hardware, software, or acombination of hardware and software. The actual specialized controlhardware or software code used to implement these systems and/or methodsis not limiting of the implementations. Thus, the operation and behaviorof the systems and/or methods are described herein without reference tospecific software code, it being understood that software and hardwarecan be designed to implement the systems and/or methods based on thedescription herein.

Some non-limiting embodiments or aspects are described herein inconnection with thresholds. As used herein, satisfying a threshold mayrefer to a value being greater than the threshold, more than thethreshold, higher than the threshold, greater than or equal to thethreshold, less than the threshold, fewer than the threshold, lower thanthe threshold, less than or equal to the threshold, equal to thethreshold, etc.

In existing systems, objects in an AV map are produced manually by ahuman operator identifying, drawing, and/or labeling roadways and/orcrosswalks in a map of the geographic location. For example, in amanually produced AV map, crosswalks are hand-drawn and manuallyestablished by using image data and map data that has been provided bythe AV map. Crosswalks are determined manually by searchingintersections or other common locations of the map to observe shapes andstyles associated with a crosswalk in the map data (e.g., a map based onmap data, etc.) associated with a crosswalk.

However, such locations in the map data may be incomplete, inaccurate,and/or insufficient to determine the location of a crosswalk. Also,crosswalks can be difficult to locate manually and/or use a timeintensive process requiring specialists (e.g., a specialist involved inthe production of a map, etc.) to find and draw (e.g., label, annotate,position, orient, etc.) crosswalks in a map. For example, specialistsmay examine an area of a map searching for indications (e.g., one ormore lines, one or more road edges, one or more boundary angles, acenterline, a sign, a speed limit, etc.) in the map data associated withthe area that may potentially provide help in determining a shape andposition of a crosswalk (e.g., a crosswalk polygon, etc.), whichpresents a difficulty associated with parsing through many values and/orvisually obscure representations of the area. Many times working withpoor representations, manually determining a crosswalk is also timeconsuming as a specialist must judge if an area of the map includescharacteristics that are both representative and sufficient to encodethe crosswalk into the map by looking at changes in the characteristicsin the AV map (e.g., changes in the angle of a crosswalk, etc.). Forexample, determining an angle, a boundary, or an edge of a crosswalkmanually are processing-intensive and inaccurate.

One of the aspects that makes AV maps particularly difficult to draw(e.g., to create, to generate, etc.) is the extent to which the AV mapmust accurately represent the position of an object (e.g., a crosswalk,a boundary, etc.), a shape, and an orientation in the map. An AV may notknow (e.g., may not obtain information to determine, etc.) when andwhere a crosswalk exists relative to a roadway in a geographic locationwhen an AV map includes manual designations of crosswalks thatinaccurately represent the crosswalk and/or may not determine withsufficient speed, accuracy, and/or precision (e.g., within a thresholddistance, etc.) whether a crosswalk may cause or create a safety issueunless the crosswalk is accurately positioned and represented in the AVmap with a proper orientation in the road or intersection. In addition,an AV using the AV map may not be able to travel in a roadway through ageographical location having a crosswalk if the AV is not able todetermine (e.g., read, etc.) where the crosswalk is located in aroadway, and/or if the position of the crosswalk is based on anincorrect judgment and/or includes an inaccurate representation (e.g., ahole in the boundary, abbreviated boundaries, etc.) in the AV map.Further, generating an AV map that includes manual designations ofcrosswalks in a geographic location may consume a relatively largeamount of network and/or processing resources and a relatively largeamount of time, as an AV map that includes manual designations for eachcrosswalk in a geographic location may not be sufficiently generatedbased on a lack of network and/or processing resources, a lack of timeto generate the AV map, and/or a lack of data to generate the AV map.Still further, map data may be insufficient or deficient, such that acrosswalk may not be determined (e.g., not accurately and/or efficientlydetermined) and/or created, and crosswalks that are created may notaccurately and/or efficiently represent the crosswalk in the AV map. Asan example, if a crosswalk is improperly or inaccurately identified in ageographic location, autonomous travel may not be available to properlycontrol an AV (e.g., an AV traveling in the roadway of the geographiclocation may not accurately predict that a crosswalk exists in the path,an AV may determine a crosswalk exists in the roadway that may not berepresented in the AV map, etc.).

Non-limiting embodiments or aspects of the present disclosure aredirected to systems, devices, products, apparatuses, and/or methods forautomatically drawing crosswalks based on image data associated with oneor more traversals of a roadway by one or more vehicles and/or providingcrosswalk information in an AV map for map production. For example, amethod may include receiving, with a computing system comprising one ormore processors, image data associated with an image of a roadwayincluding a crosswalk; generating, with the computing system, aplurality of different characteristics of the image based on the imagedata, wherein the plurality of different characteristics include aclassification of one or more elements of the image, a segmentation ofthe one or more elements of the image, and one or more angles of the oneor more elements of the image with respect to a line in the roadway;determining, with the computing system, a position of the crosswalk onthe roadway based on the plurality of different characteristics, whereinthe position includes a first boundary and a second boundary of thecrosswalk in the roadway; and providing, with the computing system, mapdata associated with a map of the roadway, wherein the map data includesthe position of the crosswalk on the roadway in the map.

In this way, rather than manually determining or labeling crosswalks ina map, modeling of image data for generating crosswalk characteristicsprovides structured crosswalk information for automatically drawingimages of a crosswalk area, may more accurately and/or efficiently usecharacteristics of one or more elements in an image associated with amap to determine a first boundary, a second boundary (e.g., a first lineand a second line of a crosswalk, etc.), and/or an angle to effectivelyenable a more accurate and efficient position in a drawing of acrosswalk in a geographic location, and provides tools, structure,and/or information signals by which crosswalks may be drawn, corrected,and/or adjusted in a map. Accordingly, time-intensive mapping ofroadways (e.g., mapping of roadways with potentially obscured and/orhard to detect crosswalks, etc.) may be reduced, navigation range andsafety of AV travel may be enhanced, networking and processing time forgenerating AV maps may be reduced, and/or a time associated with humanediting of an AV map (e.g., to label, to annotate, to position, toorient, etc.) may be reduced

Referring now to FIG. 1, FIG. 1 is a diagram of an example environment100 in which devices, systems, methods, and/or products described hereinmay be implemented. As shown in FIG. 1, environment 100 includes mapgeneration system 102, autonomous vehicle 104 including vehiclecomputing system 106, and communication network 108. Systems and/ordevices of environment 100 can interconnect via wired connections,wireless connections, or a combination of wired and wirelessconnections.

In some non-limiting embodiments or aspects, map generation system 102includes one or more devices capable of obtaining image data (e.g.,images, camera images, overhead images, LIDAR images, etc.) associatedwith one or more traversals of a roadway in a geographic location by oneor more vehicles. In some non-limiting embodiments or aspects, mapgeneration system 102 includes one or more devices capable of receivingimage data associated with an image of a roadway including a crosswalk;generating a plurality of different characteristics of the image basedon the image data, wherein the plurality of different characteristicsinclude a classification of one or more elements of the image, asegmentation of the one or more elements of the image, and an angle ofthe one or more elements of the image with respect to a line in theroadway; determining a position of the crosswalk based on the pluralityof different characteristics, wherein the position includes a firstboundary and a second boundary of the crosswalk in the roadway; andproviding map data associated with a map of the roadway, wherein the mapdata includes the position of the crosswalk on the roadway in the map.For example, map generation system 102 can include one or more computingsystems including one or more processors (e.g., one or more servers,etc.).

In some non-limiting embodiments or aspects, map generation system 102includes one or more devices capable of obtaining image data (e.g.,images, camera images, overhead images, a LIDAR point cloud, a RADARimage, etc.) and/or sensor data that can be converted to image data(e.g., accelerometers, gyroscopes, etc.) associated with one or moretraversals of a roadway in a geographic location by one or morevehicles. As an example, autonomous vehicle 104 (e.g., vehicle computingsystem 106, etc.) includes one or more devices capable of obtainingsensor data associated with parameters (e.g., pitch, pitch range, yaw,yaw range, longitudinal acceleration, latitudinal speed, longitudinalspeed, latitudinal speed, etc.) to determine the motion (e.g.,positions, heading, pose, velocity, etc.) of the AV as it traverses aroute in a geographic location. All of these different parameters maypresent a different value when an AV traverses a crosswalk. While the AVis traveling, it can factor in changes in elevation, such as hills orturns, and pass them to the map generation system 102 to factor into themap data associated with an environment surrounding autonomous vehicle104 while controlling travel, operation, and/or routing of autonomousvehicle 104 based on map data, feature data, position data, state data,sensor data, motion data, test data, and/or the like.

In some non-limiting embodiments or aspects, autonomous vehicle 104(e.g., vehicle computing system 106, etc.) includes one or more devicescapable of obtaining sensor data (e.g., image data, LIDAR data, RedGreen Blue (RGB) data, etc.) associated with a detected object in anenvironment surrounding autonomous vehicle 104. For example, autonomousvehicle 104 can include one or more computing systems including one ormore processors (e.g., one or more servers, etc.) and one or moredevices capable of obtaining sensor data (e.g., one or more wide anglecameras, LIDAR, RADAR, etc.). As an example, autonomous vehicle 104(e.g., vehicle computing system 106, etc.) includes one or more devicescapable of obtaining sensor data associated with parameters to determinethe motion (e.g., positions, heading, pose, velocity, etc.) of the AV asit traverses a route in a geographic location. All of these differentparameters present a different value when an AV traverses a crosswalk.While the AV is traveling, it can obtain images of an environmentsurrounding an AV and pass image data to the map generation system 102to factor into the map data associated with an environment surroundingautonomous vehicle 104 while controlling travel, operation, and/orrouting of autonomous vehicle 104 based on map data, feature data,position data, state data, sensor data, motion data, test data, and/orthe like. In such an example, the one or more devices may obtain sensordata associated with detecting roadway features including a crosswalk inan environment surrounding autonomous vehicle 104, for example, bycontrolling the one or more devices of autonomous vehicle 104 to detectimage data in a geographic location based on map data, feature data,position data, state data, motion data, and/or the like. Further detailsregarding non-limiting embodiments of autonomous vehicle 104 areprovided below with regard to FIG. 2.

In some non-limiting embodiments or aspects, map generation system 102and/or autonomous vehicle 104 include one or more devices capable ofreceiving, storing, and/or providing map data (e.g., map data, AV mapdata, coverage map data, hybrid map data, submap data, Uber's HexagonalHierarchical Spatial Index (H3) data, Google's S2 geometry data, etc.)associated with a map (e.g., a map, a submap, an AV map, a coverage map,a hybrid map, an H3 cell, an S2 cell, etc.) of a geographic location(e.g., a country, a state, a city, a portion of a city, a township, aportion of a township, etc.). For example, maps can be used for routingautonomous vehicle 104 on a roadway specified in the map.

In some non-limiting embodiments or aspects, a road refers to a paved orotherwise improved path between two places that allows for travel by avehicle (e.g., autonomous vehicle 104, etc.). Additionally oralternatively, a road includes a roadway and a sidewalk in proximity to(e.g., adjacent, near, next to, touching, etc.) the roadway. In somenon-limiting embodiments or aspects, a roadway includes a portion ofroad on which a vehicle is intended to travel and is not restricted by aphysical barrier or a separation so that the vehicle is able to travellaterally. Additionally or alternatively, a roadway includes one or morelanes, such as a travel lane (e.g., a lane upon which a vehicle travels,a traffic lane, etc.), a parking lane (e.g., a lane in which a vehicleparks), a bicycle lane (e.g., a lane in which a bicycle travels), aturning lane (e.g., a lane in which a vehicle turns from), a crossinglane (e.g., a crosswalk in which a pedestrian and/or other terrestrialanimal crosses, etc.) and/or the like. In some non-limiting embodimentsor aspects, a roadway is connected to another roadway, for example, alane of a roadway is connected to another lane of the roadway, and/or alane of the roadway is connected to a lane of another roadway.

In some non-limiting embodiments or aspects, a roadway is associatedwith map data that defines one or more attributes of (e.g., metadataassociated with) the roadway (e.g., attributes of a roadway in ageographic location, attributes of a segment of a roadway, attributes ofa lane of a roadway, attributes of an edge of a roadway, attributes of adriving path of a roadway, etc.). In some non-limiting embodiments oraspects, an attribute of a roadway includes an edge of a road (e.g., alocation of an edge of a road, a distance of a location from an edge ofa road, an indication whether a location is within an edge of a road,etc.), an intersection, connection, or link of a road with another road,a roadway of a road, a distance of a roadway from another roadway (e.g.,a distance of an end of a lane and/or a roadway segment or extent to anend of another lane and/or an end of another roadway segment or extent,etc.), a lane of a roadway of a road (e.g., a travel lane of a roadway,a parking lane of a roadway, a turning lane of a roadway, lane markings,a direction of travel in a lane of a roadway, etc.), a centerline of aroadway (e.g., an indication of a centerline path in at least one laneof the roadway for controlling autonomous vehicle 104 during operation(e.g., following, traveling, traversing, routing, etc.) on a drivingpath, etc.), a driving path of a roadway (e.g., one or more trajectoriesthat autonomous vehicle 104 can traverse in the roadway and anindication of the location of at least one feature in the roadway alateral distance from the driving path, etc.), one or more objects(e.g., a vehicle, a crosswalk, an edge of a road, a traffic face,vegetation, a pedestrian, a structure, a building, a sign, a lamppost,signage, a traffic sign, a bicycle, a railway track, a hazardous object,etc.) in proximity to and/or within a road (e.g., objects in proximityto the edges of a road and/or within the edges of a road), a sidewalk ofa road, and/or the like. In some non-limiting embodiments or aspects, amap of a geographic location includes one or more routes that includeone or more roadways. In some non-limiting embodiments or aspects, mapdata associated with a map of the geographic location associates eachroadway of the one or more roadways with an indication of whether anautonomous vehicle 104 can travel on that roadway.

In some non-limiting embodiments or aspects, a driving path may includefeature data associated with features of the roadway (e.g., a section ofcurb, a marker, an object, etc.) for controlling autonomous vehicle 104to autonomously determine objects in the roadway and/or feature dataassociated with longitudinal regions (e.g., a lane extending from thefront or back edge of an AV, etc.) and with lateral regions (e.g., leftand right edges of a lane in the roadway, etc.) of the driving path. Asan example, a driving path includes a trajectory (e.g., a spline, apolyline, etc.), and a location of features (e.g., a portion of thefeature, a section of the feature, etc.) in the roadway, with a link fortransitioning between an entry point and an end point of the drivingpath based on at least one of heading information, curvatureinformation, vehicle pose, acceleration information, and/or the like,and intersections with features in the roadway (e.g., real objects,paint markers, curbs, other lane paths) of a lateral region (e.g.,polygon) projecting from the path with objects of interest.

In some non-limiting embodiments or aspects, communication network 108includes one or more wired and/or wireless networks. For example,communication network 108 includes a cellular network (e.g., a long-termevolution (LTE) network, a third generation (3G) network, a fourthgeneration (4G) network, a fifth generation (5G) network, a codedivision multiple access (CDMA) network, etc.), a public land mobilenetwork (PLMN), a local area network (LAN), a wide area network (WAN), ametropolitan area network (MAN), a telephone network (e.g., the publicswitched telephone network (PSTN), a private network, an ad hoc network,an intranet, the Internet, a fiber optic-based network, a cloudcomputing network, and/or the like, and/or a combination of these orother types of networks.

The number and arrangement of systems, devices, and networks shown inFIG. 1 are provided as an example. There can be additional systems,devices and/or networks, fewer systems, devices, and/or networks,different systems, devices, and/or networks, or differently arrangedsystems, devices, and/or networks than those shown in FIG. 1.Furthermore, two or more systems or devices shown in FIG. 1 can beimplemented within a single system or a single device, or a singlesystem or a single device shown in FIG. 1 can be implemented asmultiple, distributed systems or devices. Additionally, oralternatively, a set of systems or a set of devices (e.g., one or moresystems, one or more devices) of environment 100 can perform one or morefunctions described as being performed by another set of systems oranother set of devices of environment 100.

Referring now to FIG. 2, FIG. 2 is a diagram of a non-limitingembodiment or aspect of a system 200 for controlling autonomous vehicle104. As shown in FIG. 2, vehicle computing system 106 includes vehiclecommand system 218, perception system 228, prediction system 230, motionplanning system 232, local route interpreter 234, and map geometrysystem 236 that cooperate to perceive a surrounding environment ofautonomous vehicle 104, determine a motion plan of autonomous vehicle104 based on the perceived surrounding environment, and control themotion (e.g., the direction of travel) of autonomous vehicle 104 basedon the motion plan.

In some non-limiting embodiments or aspects, vehicle computing system106 is connected to or includes positioning system 208. In somenon-limiting embodiments or aspects, positioning system 208 determines aposition (e.g., a current position, a past position, etc.) of autonomousvehicle 104. In some non-limiting embodiments or aspects, positioningsystem 208 determines a position of autonomous vehicle 104 based on aninertial sensor, a satellite positioning system, an IP address (e.g., anIP address of autonomous vehicle 104, an IP address of a device inautonomous vehicle 104, etc.), triangulation based on network components(e.g., network access points, cellular towers, Wi-Fi access points,etc.), and/or proximity to network components, and/or the like. In somenon-limiting embodiments or aspects, the position of autonomous vehicle104 is used by vehicle computing system 106.

In some non-limiting embodiments or aspects, vehicle computing system106 receives sensor data from one or more sensors 210 that are coupledto or otherwise included in autonomous vehicle 104. For example, one ormore sensors 210 includes a Light Detection and Ranging (LIDAR) system,a Radio Detection and Ranging (RADAR) system, one or more cameras (e.g.,visible spectrum cameras, infrared cameras, etc.), one more devices forcollecting positioning and pose data (e.g., pitch, latitude, longitude,altitude, etc.) and/or the like. In some non-limiting embodiments oraspects, one or more sensors 210 collect sensor data that includes datathat describes a location (e.g., in three-dimensional space relative toautonomous vehicle 104) of points that correspond to a location of theautonomous vehicle 104. In some non-limiting embodiments or aspects, thesensor data includes data that describes a location of objects withinthe surrounding environment of autonomous vehicle 104. In somenon-limiting embodiments or aspects, one or more sensors 210 collectsensor data that includes data that describes a location (e.g., inthree-dimensional space relative to autonomous vehicle 104) of pointsthat correspond to objects within the surrounding environment ofautonomous vehicle 104.

In some non-limiting embodiments or aspects, the sensor data includes alocation (e.g., a location in three-dimensional space relative to theLIDAR system) of a number of points (e.g., a point cloud) thatcorrespond to objects that have reflected a ranging laser. In somenon-limiting embodiments or aspects, the LIDAR system measures distancesby measuring a Time of Flight (TOF) that a short laser pulse takes totravel from a sensor of the LIDAR system to an object and back, and theLIDAR system calculates the distance of the object to the LIDAR systembased on the known speed of light. In some non-limiting embodiments oraspects, map data includes LIDAR point cloud maps associated with ageographic location (e.g., a location in three-dimensional spacerelative to the LIDAR system of a mapping vehicle) of a number of points(e.g., a point cloud) that correspond to objects that have reflected aranging laser of one or more mapping vehicles at the geographiclocation. As an example, a map can include a LIDAR point cloud layerthat represents objects and distances between objects in the geographiclocation of the map.

In some non-limiting embodiments or aspects, the sensor data includes alocation (e.g., a location in three-dimensional space relative to theRADAR system) of a number of points that correspond to objects that havereflected a ranging radio wave. In some non-limiting embodiments oraspects, radio waves (e.g., pulsed radio waves or continuous radiowaves) transmitted by the RADAR system can reflect off an object andreturn to a receiver of the RADAR system. The RADAR system can thendetermine information about the object's location and/or speed. In somenon-limiting embodiments or aspects, the RADAR system providesinformation about the location and/or the speed of an object relative tothe RADAR system based on the radio waves.

In some non-limiting embodiments or aspects, image processing techniques(e.g., range imaging techniques, as an example, structure from motion,structured light, stereo triangulation, etc.) can be performed by system200 to identify a location (e.g., in three-dimensional space relative tothe one or more cameras) of a number of points that correspond toobjects that are depicted in images captured by one or more cameras.Other sensors can identify the location of points that correspond toobjects as well.

In some non-limiting embodiments or aspects, map database 214 providesdetailed information associated with the map, features of the roadway inthe geographic location, and information about the surroundingenvironment of autonomous vehicle 104 for autonomous vehicle 104 to usewhile driving (e.g., traversing a route, planning a route, determining amotion plan, controlling autonomous vehicle 104, etc.).

In some non-limiting embodiments or aspects, vehicle computing system106 receives a vehicle pose from localization system 216 based on one ormore sensors 210 that are coupled to or otherwise included in autonomousvehicle 104. In some non-limiting embodiments or aspects, localizationsystem 216 includes a LIDAR localizer, a low quality pose localizer,and/or a pose filter. For example, the localization system 216 uses apose filter that receives and/or determines one or more valid poseestimates (e.g., not based on invalid position data, etc.) from theLIDAR localizer and/or the low quality pose localizer, for determining amap-relative vehicle pose. For example, low quality pose localizerdetermines a low quality pose estimate in response to receiving positiondata from positioning system 208 for operating (e.g., routing,navigating, controlling, etc.) autonomous vehicle 104 under manualcontrol (e.g., in a coverage lane, on a coverage driving path, etc.). Insome non-limiting embodiments or aspects, LIDAR localizer determines aLIDAR pose estimate in response to receiving sensor data (e.g., LIDARdata, RADAR data, etc.) from sensors 210 for operating (e.g., routing,navigating, controlling, etc.) autonomous vehicle 104 under autonomouscontrol (e.g., in an AV lane, on an AV driving path, etc.).

In some non-limiting embodiments or aspects, vehicle command system 218includes vehicle commander system 220, navigator system 222, path and/orlane associator system 224, and local route generator 226 that cooperateto route and/or navigate autonomous vehicle 104 in a geographiclocation. In some non-limiting embodiments or aspects, vehicle commandersystem 220 provides tracking of a current objective of autonomousvehicle 104, such as a current service, a target pose, a coverage plan(e.g., development testing, etc.), and/or the like. In some non-limitingembodiments or aspects, navigator system 222 determines and/or providesa route plan (e.g., a route between a starting location or a currentlocation and a destination location, etc.) for autonomous vehicle 104based on a current state of autonomous vehicle 104, map data (e.g., lanegraph, driving paths, etc.), and one or more vehicle commands (e.g., atarget pose). For example, navigator system 222 determines a route plan(e.g., a plan, a re-plan, a deviation from a route plan, etc.) includingone or more lanes (e.g., current lane, future lane, etc.) and/or one ormore driving paths (e.g., a current driving path, a future driving path,etc.) in one or more roadways that autonomous vehicle 104 can traverseon a route to a destination location (e.g., a target location, a tripdrop-off location, etc.).

In some non-limiting embodiments or aspects, navigator system 222determines a route plan based on one or more lanes and/or one or moredriving paths received from path and/or lane associator system 224. Insome non-limiting embodiments or aspects, path and/or lane associatorsystem 224 determines one or more lanes and/or one or more driving pathsof a route in response to receiving a vehicle pose from localizationsystem 216. For example, path and/or lane associator system 224determines, based on the vehicle pose, that autonomous vehicle 104 is ona coverage lane and/or a coverage driving path, and in response todetermining that autonomous vehicle 104 is on the coverage lane and/orthe coverage driving path, determines one or more candidate lanes (e.g.,routable lanes, etc.) and/or one or more candidate driving paths (e.g.,routable driving paths, etc.) within a distance of the vehicle poseassociated with autonomous vehicle 104. For example, path and/or laneassociator system 224 determines, based on the vehicle pose, thatautonomous vehicle 104 is on an AV lane and/or an AV driving path, andin response to determining that autonomous vehicle 104 is on the AV laneand/or the AV driving path, determines one or more candidate lanes(e.g., routable lanes, etc.) and/or one or more candidate driving paths(e.g., routable driving paths, etc.) within a distance of the vehiclepose associated with autonomous vehicle 104. In some non-limitingembodiments or aspects, navigator system 222 generates a cost functionfor each of the one or more candidate lanes and/or the one or morecandidate driving paths that autonomous vehicle 104 may traverse on aroute to a destination location. For example, navigator system 222generates a cost function that describes a cost (e.g., a cost over atime period) of following (e.g., adhering to) one or more lanes and/orone or more driving paths that may be used to reach the destinationlocation (e.g., a target pose, etc.).

In some non-limiting embodiments or aspects, local route generator 226generates and/or provides route options that may be processed andcontrol travel of autonomous vehicle 104 on a local route. For example,navigator system 222 may configure a route plan, and local routegenerator 226 may generate and/or provide one or more local routes orroute options for the route plan. For example, the route options mayinclude one or more options for adapting the motion of the AV to one ormore local routes in the route plan (e.g., one or more shorter routeswithin a global route between the current location of the AV and one ormore exit locations located between the current location of the AV andthe destination location of the AV, etc.). In some non-limitingembodiments or aspects, local route generator 226 may determine a numberof route options based on a predetermined number, a current location ofthe AV, a current service of the AV, and/or the like.

In some non-limiting embodiments or aspects, perception system 228detects and/or tracks objects (e.g., bumps, depressions, vehicles,pedestrians, bicycles, and the like) that are proximate to (e.g., inproximity to the surrounding environment of) autonomous vehicle 104 overa time period. In some non-limiting embodiments or aspects, perceptionsystem 228 can retrieve (e.g., obtain) map data from map database 214that provides detailed information about the surrounding environment ofautonomous vehicle 104.

In some non-limiting embodiments or aspects, perception system 228determines one or more objects that are proximate to autonomous vehicle104 based on sensor data received from one or more sensors 210 and/ormap data from map database 214. For example, perception system 228determines, for the one or more objects that are proximate, state dataassociated with a state of such an object. In some non-limitingembodiments or aspects, the state data associated with an objectincludes data associated with a location of the object (e.g., aposition, a current position, an estimated position, etc.), dataassociated with a speed of the object (e.g., a magnitude of velocity ofthe object), data associated with a direction of travel of the object(e.g., a heading, a current heading, etc.), data associated with anacceleration rate of the object (e.g., an estimated acceleration rate ofthe object, etc.), data associated with an orientation of the object(e.g., a current orientation, etc.), data associated with a size of theobject (e.g., a size of the object as represented by a bounding shape,such as a bounding polygon or polyhedron, a footprint of the object,etc.), data associated with a type of the object (e.g., a class of theobject, an object with a type of vehicle, an object with a type ofpedestrian, an object with a type of bicycle, etc.), and/or the like.

In some non-limiting embodiments or aspects, perception system 228determines state data for an object over a number of iterations ofdetermining state data. For example, perception system 228 updates thestate data for each object of a plurality of objects during eachiteration.

In some non-limiting embodiments or aspects, prediction system 230receives the state data associated with one or more objects fromperception system 228. Prediction system 230 predicts one or more futurelocations for the one or more objects based on the state data. Forexample, prediction system 230 predicts the future location of eachobject of a plurality of objects within a time period (e.g., 5 seconds,10 seconds, 20 seconds, etc.). In some non-limiting embodiments oraspects, prediction system 230 predicts future movement of an objectaccording to a probable class of the objects, one or more conditions,one or more observed locations of the object, and/or map data. Forexample, prediction system 230 predicts a future movement of an objectaccording to one or more prior probability distributions including moverclassifications, one or more motion paths, and/or one or more drivingpaths based on detection of the object and/or movement of the object.

In some non-limiting embodiments or aspects, prediction system 230predicts that an object will adhere to the object's direction of travelaccording to the speed of the object. In some non-limiting embodimentsor aspects, prediction system 230 uses machine learning techniques ormodeling techniques to make a prediction based on state data associatedwith an object.

In some non-limiting embodiments or aspects, prediction system 230predicts future movement of an object according to one or more observedlocations of a static object in the roadway (e.g., a crosswalk, etc.),and/or map data.

In some non-limiting embodiments or aspects, motion planning system 232determines a motion plan for autonomous vehicle 104 based on aprediction of a location associated with an object provided byprediction system 230 and/or based on state data associated with theobject provided by perception system 228. For example, motion planningsystem 232 determines a motion plan (e.g., an optimized motion plan) forautonomous vehicle 104 that causes autonomous vehicle 104 to travelrelative to the object based on the prediction of the location for theobject provided by prediction system 230 and/or the state dataassociated with the object provided by perception system 228.

In some non-limiting embodiments or aspects, motion planning system 232receives a route plan as a command from navigator system 222. In somenon-limiting embodiments or aspects, motion planning system 232determines a cost function for one or more motion plans of a route forautonomous vehicle 104 based on the locations and/or predicted locationsof one or more objects. For example, motion planning system 232determines the cost function that describes a cost (e.g., a cost over atime period) of following (e.g., adhering to) a motion plan (e.g., aselected motion plan, an optimized motion plan, etc.). In somenon-limiting embodiments or aspects, the cost associated with the costfunction increases and/or decreases based on autonomous vehicle 104deviating from a motion plan (e.g., a selected motion plan, an optimizedmotion plan, a preferred motion plan, etc.). For example, the costassociated with the cost function increases and/or decreases based onautonomous vehicle 104 deviating from the motion plan to avoid acollision with an object.

In some non-limiting embodiments or aspects, motion planning system 232determines a cost of following a motion plan. For example, motionplanning system 232 determines a motion plan for autonomous vehicle 104based on one or more cost functions. In some non-limiting embodiments oraspects, motion planning system 232 determines a motion plan (e.g., aselected motion plan, an optimized motion plan, a preferred motion plan,etc.) that minimizes a cost function. In some non-limiting embodimentsor aspects, motion planning system 232 provides a motion plan to vehiclecontrols 240 (e.g., a device that controls acceleration, a device thatcontrols steering, a device that controls braking, an actuator thatcontrols gas flow, etc.) to implement the motion plan.

In some non-limiting embodiments or aspects, motion planning system 232can determine a motion plan for autonomous vehicle 104 based on thepredicted future locations and/or motion paths for the object providedby the prediction system 230 and/or the state data for the objectprovided by the perception system 228. Stated differently, giveninformation about current locations of objects and/or predicted futurelocations and/or motion paths of objects, the motion planning system 232can determine a motion plan for autonomous vehicle 104 that navigatesautonomous vehicle 104 relative to the objects at such locationsaccording to the cost function.

In some non-limiting embodiments or aspects, motion planning system 232communicates with local route interpreter 234 and map geometry system236. In some non-limiting embodiments or aspects, local routeinterpreter 234 may receive and/or process route options from localroute generator 226. For example, local route interpreter 234 maydetermine a new or updated route for travel of autonomous vehicle 104.As an example, one or more lanes and/or one or more driving paths in alocal route may be determined by local route interpreter 234 and mapgeometry system 236. For example, local route interpreter 234 candetermine a route option and map geometry system 236 determines one ormore lanes and/or one or more driving paths in the route option forcontrolling the motion of autonomous vehicle 104.

Referring now to FIG. 3, FIG. 3 is a diagram of example components of adevice 300. Device 300 can correspond to one or more devices of mapgeneration system 102 and/or one or more devices of (e.g., one or moredevices of a system of) autonomous vehicle 104. In some non-limitingembodiments or aspects, one or more devices of map generation system 102and/or one or more devices of (e.g., one or more devices of a system of)autonomous vehicle 104 can include at least one device 300 and/or atleast one component of device 300. As shown in FIG. 3, device 300includes bus 302, processor 304, memory 306, storage component 308,input component 310, output component 312, and communication interface314.

Bus 302 includes a component that permits communication among thecomponents of device 300. In some non-limiting embodiments or aspects,processor 304 is implemented in hardware, firmware, or a combination ofhardware and software. For example, processor 304 includes a processor(e.g., a central processing unit (CPU), a graphics processing unit(GPU), an accelerated processing unit (APU), etc.), a microprocessor, adigital signal processor (DSP), and/or any processing component (e.g., afield-programmable gate array (FPGA), an application-specific integratedcircuit (ASIC), etc.) that can be programmed to perform a function.Memory 306 includes a random access memory (RAM), a read only memory(ROM), and/or another type of dynamic or static storage device (e.g.,flash memory, magnetic memory, optical memory, etc.) that storesinformation and/or instructions for use by processor 304.

Storage component 308 stores information and/or software related to theoperation and use of device 300. For example, storage component 308includes a hard disk (e.g., a magnetic disk, an optical disk, amagneto-optic disk, a solid state disk, etc.), a compact disc (CD), adigital versatile disc (DVD), a floppy disk, a cartridge, a magnetictape, and/or another type of computer-readable medium, along with acorresponding drive.

Input component 310 includes a component that permits device 300 toreceive information, such as via user input (e.g., a touch screendisplay, a keyboard, a keypad, a mouse, a button, a switch, amicrophone, etc.). Additionally, or alternatively, input component 310includes a sensor for sensing information (e.g., a global positioningsystem (GPS) component, an accelerometer, a gyroscope, an actuator,etc.). Output component 312 includes a component that provides outputinformation from device 300 (e.g., a display, a speaker, one or morelight-emitting diodes (LEDs), etc.).

Communication interface 314 includes a transceiver-like component (e.g.,a transceiver, a separate receiver and transmitter, etc.) that enablesdevice 300 to communicate with other devices, such as via a wiredconnection, a wireless connection, or a combination of wired andwireless connections. Communication interface 314 can permit device 300to receive information from another device and/or provide information toanother device. For example, communication interface 314 includes anEthernet interface, an optical interface, a coaxial interface, aninfrared interface, a radio frequency (RF) interface, a universal serialbus (USB) interface, a Wi-Fi interface, a cellular network interface,and/or the like.

Device 300 can perform one or more processes described herein. Device300 can perform these processes based on processor 304 executingsoftware instructions stored by a computer-readable medium, such asmemory 306 and/or storage component 308. A computer-readable medium(e.g., a non-transitory computer-readable medium) is defined herein as anon-transitory memory device. A memory device includes memory spacelocated inside of a single physical storage device or memory spacespread across multiple physical storage devices.

Software instructions can be read into memory 306 and/or storagecomponent 308 from another computer-readable medium or from anotherdevice via communication interface 314. When executed, softwareinstructions stored in memory 306 and/or storage component 308 causeprocessor 304 to perform one or more processes described herein.Additionally, or alternatively, hardwired circuitry can be used in placeof or in combination with software instructions to perform one or moreprocesses described herein. Thus, embodiments described herein are notlimited to any specific combination of hardware circuitry and software.

Memory 306 and/or storage component 308 may include data storage or oneor more data structures (e.g., a database, etc.). Device 300 may becapable of receiving information from, storing information in,communicating information to, or searching information stored in thedata storage or one or more data structures in memory 306 and/or storagecomponent 308. In some non-limiting embodiments or aspects, theinformation may include data (e.g., sensor data, log data, one or moreprior probability distributions, etc.) associated with one or moretraversals of a roadway by one or more vehicles.

The number and arrangement of components shown in FIG. 3 are provided asan example. In some non-limiting embodiments or aspects, device 300includes additional components, fewer components, different components,or differently arranged components than those shown in FIG. 3.Additionally, or alternatively, a set of components (e.g., one or morecomponents) of device 300 can perform one or more functions described asbeing performed by another set of components of device 300.

Referring now to FIG. 4, FIG. 4 is a flowchart of a non-limitingembodiment of a process 400 for determining a crosswalk. In somenon-limiting embodiments or aspects, one or more of the steps of process400 are performed (e.g., completely, partially, etc.) by map generationsystem 102 (e.g., one or more devices of map generation system 102,etc.). In some non-limiting embodiments or aspects, one or more of thesteps of process 400 are performed (e.g., completely, partially, etc.)by another device or a group of devices separate from or including mapgeneration system 102, such as autonomous vehicle 104 (e.g., one or moredevices of autonomous vehicle 104, etc.).

As shown in FIG. 4, at step 402, process 400 includes receiving imagedata associated with an image of a roadway including a crosswalk. Forexample, map generation system 102 receives image data associated withan image of a roadway including a crosswalk. As an example, mapgeneration system 102 receives (e.g., obtains, retrieves, etc.) imagedata associated with an image of a roadway including a crosswalk.

In some non-limiting embodiments or aspects, image data includes ahistogram (e.g., a two dimensional histogram, a two dimensionalhistogram of cloud points representing a feature of a road, a normalizedhistogram, a feature map, etc.) associated with a feature of a road(e.g., a crosswalk, an intersection, etc.) included in the geographicarea. In some non-limiting embodiments or aspects, a histogram mayinclude a value (e.g., a number between 0-255) of a color of eachelement (e.g., a pixel) in an image of the geographic area. In somenon-limiting embodiments or aspects, the value of the color may varyamong shades of red, green, and blue.

In some non-limiting embodiments or aspects, image data may include ahistogram associated with an object in proximity to and/or within aroad. Additionally or alternatively, image data may include a histogramassociated with a road edge of a road (e.g., a histogram of a distanceof a location from a road edge of a road, a histogram of a locationwithin and/or outside of a road edge of a road, etc.), a histogram of aroadway of a road (e.g., a histogram of a lane of a roadway of a road, ahistogram of a travel lane of a roadway, a histogram of a parking laneof a roadway, etc.), and/or a histogram of a crosswalk of a road. Insome non-limiting embodiments or aspects, a histogram associated with anobject in proximity to and/or within the road may indicate a probabilitymass concentrated around the object.

Additionally or alternatively, image data may include a histogram of anintensity of a color (e.g., black, white, etc.) in the image of thegeographic area. In some non-limiting embodiments or aspects, mapgeneration system 102 may determine a homogenous area of a geographicarea based on the histogram of the intensity of the color. For example,map generation system 102 may determine a lane of a roadway (e.g., aparking lane, a travel lane, etc.) in the geographic area based on theintensity of the color being the same in the portion of the geographicarea that includes the lane. Additionally or alternatively, image datamay include a top down camera image (e.g., a synthesized Red Green Blue(RGB) image) of the geographic area. In some non-limiting embodiments oraspects, map generation system 102 may determine a homogenous area of ageographic area based on the top down camera image. For example, mapgeneration system 102 may determine a crosswalk of a roadway in thegeographic area based on the color being the same in the top down cameraimage of the geographic area that includes the crosswalk.

In some non-limiting embodiments or aspects, map generation system 102obtains image data that includes parameter information associated withone or more parameters collected as a vehicle transitions through asequence of poses while traversing a roadway in a geographic location.As an example, map generation system 102 obtains image data collected byautonomous vehicles and/or non-autonomous, human controlled vehicles asthe vehicles travel on a roadway in a geographic region. For example,sensor data may be collected by vehicles (e.g., a map collectionvehicle, etc.) during travel, specifically for collecting data formapping and/or during normal autonomy operations (e.g., non-mappingspecific, etc.). As an example, vehicle computing system 106 ofautonomous vehicle 104 can obtain sensor data that has been stored invehicle logs as autonomous vehicle 104 traverses a roadway. The sensordata in the vehicle logs can be stored locally and/or sent to mapgeneration system 102 to be processed (e.g., stored, chunked,aggregated, etc.), and include one or more parameters, such as, forexample, x-value coordinates (e.g., a latitude, etc.), y-valuecoordinates (e.g., a longitude, etc.), z-value coordinates (e.g., anelevation, height, or altitude relative to the ground and/or sea level,etc.), heading (e.g., a yaw, etc.), pitch of the vehicle (e.g., a pitchangle, grade, or incline that is positive for upward movement of thevehicle nose and negative for downward movement of the vehicle nose,etc.), pitch rate, longitudinal speed, longitudinal acceleration, yaw,yaw rate, latitudinal speed, latitudinal acceleration, roll, roll rate,and/or the like.

Referring now to FIG. 5, FIG. 5 shows an example of a diagram of a modelfor receiving image data (e.g., input data, input images, etc.) andoutputting one or more characteristics of the input data. For example,input images include LIDAR data (e.g., LIDAR points, etc.) and cameraimages that are projected onto the ground to produce input images from abird's eye view. As an example, map generation system 102 uses a trainedconvolutional neural network (CNN) to generate (e.g., produce, etc.) oneor more feature maps, (e.g., three feature maps corresponding tosegmentation, classification, and alignment, etc.). For example, mapgeneration system 102 concatenates the input images together to form a 4channel input, with 3 dimensions for RGB and one for LIDAR intensity. Insome non-limiting embodiments or aspects, map generation system 102processes the image data associated with the image of the roadwayincluding the crosswalk using the trained convolutional neural networkto generate a plurality of images associated with the plurality ofdifferent characteristics of the image.

In some non-limiting embodiments or aspects, map generation system 102uses the one or more feature maps output by the CNN to performinference. For example, map generation system 102 provides the imagedata associated with a crosswalk to a convolutional neural network. Insome non-limiting embodiments or aspects, a map is used which providesroad centerlines and intersection polygons. In some non-limitingembodiments or aspects, map generation system 102 performs a structuredprediction to determine two boundaries x1 and x2 (e.g., optimalboundaries, best boundaries, etc.) along with an angle β (e.g., anoptimal angle, a best angle, etc.) by maximizing an optimizationfunction (e.g., a structured energy function, etc.).

In some non-limiting embodiments or aspects, map generation system 102overlays the image maps on road centerlines and intersection polygonsthat are typically available in publicly available maps such as theOpenStreetMap (OSM) project to parameterize a crosswalk in a structuredway, where a crosswalk is associated with a correct topology and shape.As an example, map generation system 102 determines where a crosswalkoccurs (e.g., an intersection, a crossing lane, etc.) given thetopological graphs of OSM.

With continued reference to FIG. 4, at step 404, process 400 includesgenerating different characteristics of the image based on image dataincluding a classification, a segmentation, and an angle with respect toa line in the roadway. For example, map generation system 102 generatesdifferent characteristics of the image based on image data including aclassification, a segmentation, and an angle with respect to a line inthe roadway. As an example, map generation system 102 generates aplurality of different characteristics of the image based on the imagedata, wherein the plurality of different characteristics includes aclassification of one or more elements of the image, a segmentation ofthe one or more elements of the image, and an angle of the one or moreelements of the image with respect to a line in the roadway.

In some non-limiting embodiments or aspects, map generation system 102generates the plurality of different characteristics based on a machinelearning technique (e.g., a pattern recognition technique, a data miningtechnique, a heuristic technique, a supervised learning technique, anunsupervised learning technique, etc.). For example, map generationsystem 102 may generate a model (e.g., an estimator, a classifier, aprediction model, etc.) based on a machine learning algorithm (e.g., adecision tree algorithm, a gradient boosted decision tree algorithm, aneural network algorithm, a convolutional neural network algorithm,etc.). In such an example, map generation system 102 may generate theplurality of different characteristics using the prediction model. Insome non-limiting embodiments or aspects, a prediction score may includean indication whether the image (e.g., the entire image, a portion ofthe image, an element of the image, a pixel of the image, etc.) of ageographical area includes and/or is associated with one or morecrosswalk locations and/or one or more crossing lanes.

In some non-limiting embodiments or aspects, map generation system 102generates the plurality of different characteristics based on anencoder-decoder model. In some non-limiting embodiments or aspects, anarchitecture of an encoder-decoder model can include skip connectionsand residual layers, such as feature pyramid networks to output thethree feature maps (e.g., a residual learning framework to ease trainingof networks as disclosed by

Mattyus, G., Wang, S., Fidler, S., Urtasun, R. in the paper titled“Enhancing road maps by parsing aerial images around the world”, (InICCV, 2015), the entire contents of which is hereby incorporated byreference, a multi-scale, pyramidal hierarchy of a deep convolutionalnetwork as disclosed by Mattyus, G., Wang, S., Fidler, S., Urtasun, R.in “HD maps: Fine-grained road segmentation by parsing ground and aerialimages”, (In CVPR, 2016), the entire contents of which is herebyincorporated by reference, and deep neural network architecture to learnwithout an increase in the number of parameters as disclosed by Nair,V., Hinton, G. E. in “Rectified linear units improve restrictedboltzmann machines”, (In Proceedings of the 27th InternationalConference on International Conference, 2010) the entire contents ofwhich is hereby incorporated by reference).

Referring now to FIG. 6, FIG. 6 shows an example of a diagram of anetwork architecture including one or more convolutional layers, andincluding before each convolutional layer a batch normalization (e.g., anormalizing layer input as disclosed by Visin, F., Ciccone, M., Romero,A., Kastner, K., Cho, K., Bengio, Y., Matteucci, M., Courville, A. in“Reseg: A recurrent neural network-based model for semanticsegmentation”, (In CVPR, 2016) the entire contents of which is herebyincorporated by reference), followed by a ReLU non-linearity (e.g., amore powerful type of hidden unit by tying the weights and biases of aninfinite set of binary units as disclosed in Volodymyr, I., Coughlan,J., Shen, H. in “Detecting and locating crosswalks using a cameraphone”, (In CVPRW (2008), the entire contents of which is herebyincorporated by reference).

In some non-limiting embodiments or aspects, the CNN includes an encodernetwork having one or more residual blocks including at least threeconvolutional layers. In some non-limiting embodiments or aspects,artificial neurons of a first convolutional layer of the CNN areconnected only to elements in a scanning window (e.g. receptive window,etc.) of a feature map. For example, the scanning window forms a smallsquare around elements of the feature maps. To ensure the network has alarge receptive field (e.g., to handle large images, etc.), mapgeneration system 102 may provide dilated convolutions in the residualblocks (e.g., a dilated convolution operator is particularly suited todense prediction due to its ability to expand the receptive fieldwithout losing resolution or coverage as disclosed by Teo, T. A., Rau,J. Y., Chen, L. C., Liu, J. K., Hsu, W. C. in “Reconstruction of complexbuildings using lidar and 2d maps” (In Innovations in 3D Geo InformationSystems, 2006), the entire contents of which is hereby incorporated byreference). In some non-limiting embodiments or aspects, map generationsystem 102 provides nearest neighbor upsampling back to the originalimage size in the decoder network. In some non-limiting embodiments oraspects, map generation system 102 divides the output into threebranches, for example, one for each feature map.

In some non-limiting embodiments or aspects, to predict the inversedistance transform, a ReLU is used to generate non-linearity beforegenerating outputs to restrict an output (e.g., a distance signal, avalue, etc.) to a positive value. In some non-limiting embodiments oraspects, map generation system 102 generates a segmentation by applyinga SoftMax over the output to generate a probability map. In somenon-limiting embodiments or aspects, map generation system 102 generatesan angle (e.g., an alignment map, etc.) by applying a ReLU non-linearityto restrict an output (e.g., one or more angles, a value, etc.) to apositive value.

In some non-limiting embodiments or aspects, the prediction model mayuse MaxPool2d (e.g., kernel width, kernel height, stride, padding) andConv2d (e.g., kernel width, kernel height, out channels, stride,padding, dilation). For example, map generation system 102 processesartificial neuron data with one or more pooling layers of theconvolution neural network model to produce pooling layer dataassociated with a pooling layer. For example, map generation system 102may process artificial neuron data with one or more pooling layers ofthe convolution neural network model to produce pooling layer dataassociated with a pooling layer. In some cases, the map generationsystem 102 outputs to the pooling layer artificial neurons in a firstset of rows and columns from a previous convolution layer. For example,the map generation system 102 outputs an aggregated value into thepooling layer data. For example, the map generation system 102 mayaggregate by determining a max function. In such an example, featuredata located in a first set of rows and columns of the convolutionallayers may be subsampled into artificial neuron data associated with theartificial neurons of the convolution layer. For example, only the maxinput value in each receptive area is stored as pooling layer artificialneuron data. In some cases, the map generation system 102 may determinea mean function based on the feature map.

In some non-limiting embodiments or aspects, map generation system 102may generate a prediction model (e.g., a crosswalk prediction model)based on map data associated with a map of a geographical area thatincludes a crosswalk and/or a crossing lane. In some implementations,the prediction model may include a model designed to receive, as aninput, image data associated with one or more images of one or moregeographical areas that include one or more crosswalk locations and/orone or more crosswalk lanes, and provide, as an output, a prediction(e.g., a probability, a binary output, a yes-no output, a score, aprediction score, etc.) as to whether the image (e.g., the entire image,a portion of the image, an element of the image, a pixel of the image,etc.) of a geographical area includes one or more crosswalk locations.In one example, the prediction model may be designed to receive imagedata associated with an image of a geographical area, and provide anoutput that predicts whether the image includes one or more crosswalklocations (e.g., labels one or more elements associated with acrosswalk, etc.). In some non-limiting embodiments or aspects, mapgeneration system 102 may store the prediction model (e.g., for lateruse). In some non-limiting embodiments or aspects, map generation system102 may store the prediction model in a data structure (e.g., adatabase, a linked list, a tree, etc.). The data structure may belocated within map generation system 102 or exist externally, andpossibly remote from, map generation system 102.

In some non-limiting embodiments or aspects, map generation system 102may process image data to obtain training data for the prediction model.For example, map generation system 102 may process the image data tochange the image data into a format that may be analyzed (e.g., by mapgeneration system 102) to generate a prediction model. The image datathat is changed may be referred to as training data. In somenon-limiting embodiments or aspects, map generation system 102 mayprocess the image data to obtain the training data based on receivingthe image data. Additionally, or alternatively, map generation system102 may process the image data to obtain the training data based on mapgeneration system 102 receiving an indication that map generation system102 is to process the image data from a user of map generation system102, such as when map generation system 102 receives an indication tocreate a prediction model for a portion of an image of a geographicalarea corresponding to the image data.

In some non-limiting embodiments or aspects, map generation system 102trains a convolutional neural network by modifying parameters tooptimize a loss function of the convolutional neural network thatdepends on ground truth labels and one or more predictions of theplurality of different characteristics, wherein the loss functioncomprises a sum of a segmentation loss, an alignment loss, and aboundary loss.

In some non-limiting embodiments or aspects, map generation system 102may process the image data associated with the image of the roadwayincluding the crosswalk using the trained convolutional neural network,to generate a plurality of images associated with the plurality ofdifferent characteristics of the image. For example, map generationsystem 102 may process the image data by determining an image variable.An image variable may include a metric, associated with a crosswalklocation, which may be derived based on the image data. For example,image data may include an image variable that may be analyzed togenerate a prediction model. For example, the image variable may includea variable associated with feature data of a feature map. In somenon-limiting embodiments or aspects, the image variable may a variableassociated with a road edge of a road (e.g., a variable associated witha location of a road edge of a road, a variable associated with adistance of location from a road edge of a road, a variable associatedwith an indication whether a location is within a road edge of a road,etc.), a variable associated with an intersection of a road with anotherroad, a variable associated with a roadway of a road, a variableassociated with a lane of a roadway of a road (e.g., a variableassociated with a travel lane of a roadway, a variable associated with acrosswalk lane of a roadway, a variable associated with a turning laneof a roadway, a variable associated with lane markings of a lane, avariable associated with a direction of travel in a lane of a roadway,etc.), a variable associated with one or more objects in proximity toand/or within a road, a variable associated with a sidewalk of a road,and/or the like. Additionally or alternatively, the image variable mayinclude a variable associated with an intensity of a color (e.g., black,white, etc.) in the image of the geographic area. Additionally oralternatively, the image variable may include a variable associated witha top down camera image (e.g., a synthesized Red Green Blue (RGB) image)of a geographic area.

In some non-limiting embodiments or aspects, map generation system 102may analyze the training data to generate the prediction model. Forexample, map generation system 102 may use machine learning techniquesto analyze the training data to generate the prediction model. In someimplementations, generating the prediction model (e.g., based ontraining data obtained from image data, based on training data obtainedfrom historical image data) may be referred to as training theprediction model. The machine learning techniques may include, forexample, supervised and/or unsupervised techniques, such as decisiontrees (e.g., gradient boosted decision trees), logistic regressions,artificial neural networks (e.g., convolutional neural networks),Bayesian statistics, learning automata, Hidden Markov Modeling, linearclassifiers, quadratic classifiers, association rule learning, and/orthe like. In some non-limiting embodiments or aspects, the predictionmodel may include a model that is specific to a particular geographicalarea, particular image data associated with an image of a geographicalarea, and/or the like. Additionally, or alternatively, the predictionmodel may be specific to a particular user (e.g., an operator of anautonomous vehicle, an entity that operates an autonomous vehicle,etc.). In some non-limiting embodiments or aspects, map generationsystem 102 may generate one or more prediction models for one or moreoperators of one or more autonomous vehicles, a particular group ofautonomous vehicles, and/or the like.

In some non-limiting embodiments or aspects, when analyzing the trainingdata, map generation system 102 may identify one or more image variables(e.g., one or more independent image variables) as predictor variablesthat may be used to make a prediction (e.g., when analyzing the trainingdata). In some non-limiting embodiments or aspects, values of thepredictor variables may be input to the prediction model. For example,map generation system 102 may identify a subset (e.g., a proper subset)of image variables as predictor variables that may be used to accuratelypredict whether an image (e.g., a portion of the image, an element ofthe image, a pixel of the image, etc.) of a geographical area includesone or more crosswalk locations. In some non-limiting embodiments oraspects, the predictor variables may include one or more of the imagevariables, as discussed above, that have a significant impact (e.g., animpact satisfying a threshold) on a probability that the image of thegeographical area includes one or more crosswalk locations.

In some non-limiting embodiments or aspects, map generation system 102may validate the prediction model. For example, map generation system102 may validate the prediction model after map generation system 102generates the prediction model. In some non-limiting embodiments oraspects, map generation system 102 may validate the prediction modelbased on a portion of the training data to be used for validation. Forexample, map generation system 102 may partition the training data intoa first portion and a second portion, where the first portion may beused to generate the prediction model, as described above. In thisexample, the second portion of the training data (e.g., the validationdata) may be used to validate the prediction model.

In some non-limiting embodiments or aspects, map generation system 102may validate the prediction model by providing validation dataassociated with an image of a geographical area as input to theprediction model, and determining, based on an output of the predictionmodel, whether the prediction model correctly, or incorrectly, predictedthat the image of the geographical area includes one or more crosswalklocations. In some implementations, map generation system 102 mayvalidate the prediction model based on a validation threshold. Forexample, map generation system 102 may be configured to validate theprediction model when an image of a geographical area is correctlypredicted by the prediction model (e.g., when the prediction modelcorrectly predicts 50% of the validation data, 70% of the validationdata, a threshold quantity of the validation data, etc.) as includingone or more crosswalk locations.

In some non-limiting embodiments or aspects, if map generation system102 does not validate the prediction model (e.g., when a percentage ofvalidation data does not satisfy the validation threshold), then mapgeneration system 102 may generate additional prediction models.

In some non-limiting embodiments or aspects, once the prediction modelhas been validated, map generation system 102 may further train theprediction model and/or create new prediction models based on receivingnew training data. The new training data may include image dataassociated with an image of another geographical area).

In some non-limiting embodiments or aspects, map generation system 102may generate plurality of different characteristics using a model (e.g.,a convolutional neural network, an artificial neural network, a boosteddecision tree, a gradient boosted decision tree, etc.). In one example,map generation system 102 may receive one or more feature maps thatinclude feature data associated with one or more features of a road.Additionally or alternatively, map generation system 102 may perform asmoothing process on the one or more feature maps based on receiving theone or more feature maps.

In the example above, the map generation system 102 may determinefeature data associated with an element of a multi-dimensional matrix ofone or more feature maps. In some non-limiting embodiments or aspects,map generation system 102 may determine feature data associated with anelement of a first feature map and feature data associated with anelement of a second feature map. The location of the element of thefirst feature map may correspond to the location of the element of thematrix of the second feature map. For example, the coordinate of theelement (e.g., the value of the row of the matrix, the value of thecolumn of the matrix) of the first feature map may correspond to thecoordinate of the element of the second feature map. In somenon-limiting embodiments or aspects, the first feature map may have asize that is the same as or different from the size of the secondfeature map. In some non-limiting embodiments or aspects, map generationsystem 102 may determine feature data associated with an element of afeature map upon which a smoothing process has been performed andfeature data associated with an element of a feature map upon which asmoothing process has not been performed.

In some non-limiting embodiments or aspects, map generation system 102may provide the feature data associated with the element of amulti-dimensional matrix of the one or more feature maps as an input toa model. Map generation system 102 may receive the plurality ofdifferent characteristics as an output of the decision tree model.

In some non-limiting embodiments or aspects, map generation system 102may generate a map (e.g., a crosswalk prediction map) based on theplurality of different characteristics. For example, map generationsystem 102 may generate the crosswalk prediction map based on theplurality of different characteristics generated using the machinelearning technique. In some non-limiting embodiments or aspects, mapgeneration system 102 may generate an overlay that includes theplurality of different characteristics associated with one or moreelements of the map that was used to determine the plurality ofdifferent characteristics. In some non-limiting embodiments or aspects,map generation system 102 may generate the crosswalk prediction map bycombining the overlay and the map that was used to determine theplurality of different characteristics. In some non-limiting embodimentsor aspects, map generation system 102 may combine the overlay with a mapthat includes a label (e.g., a label provided by an individual) of acrosswalk location.

In some non-limiting embodiments or aspects, map generation system 102may apply a post-processing technique (e.g., spatial smoothing,thresholding, clustering, such as detecting connected components,creating convex hulls around a cluster, applying a minimum bounding box,merging areas in proximity to each other, rotating elements of thecrosswalk prediction map, applying map constraints to the crosswalkprediction map, etc.) to the crosswalk prediction map.

In some non-limiting embodiments or aspects, map generation system 102may apply a bilateral filter on a prediction score associated with anelement of the crosswalk prediction map to perform spatial smoothing.For example, map generation system 102 may apply the bilateral filterand replace the prediction score of the element with a weighted averageof prediction scores associated with elements that are in proximity tothe element upon which the bilateral filter was applied. In this way,map generation system 102 may allow easier detection of a crosswalk laneas compared to foregoing to apply a bilateral filter. Map generationsystem 102 may allow for easier detection of a crosswalk lane since thebilateral filter may preserve sharp edges between a crosswalk lane andother features of a road. In some non-limiting embodiments or aspects,the weighted average may be Gaussian-shaped based on the distance (e.g.,the Euclidean distance) between the element upon which the bilateralfilter was applied and one or more elements that are in proximity to theelement.

In some embodiments, map generation system 102 may convert a predictionscore associated with an element of a crosswalk prediction map to anassigned value (e.g., a label) by comparing the prediction score to athreshold value of a prediction score. For example, map generationsystem 102 may assign a value (e.g., 1 or 0) to the element based on theprediction score associated with the element satisfying the thresholdvalue.

In some non-limiting embodiments or aspects, map generation system 102outputs (e.g., automatically) image data based on generating the one ormore different characteristics of the image based on image data. Forexample, the map generation system 102 outputs one or more images basedon the image data, but the number of images output will be equal to thenumber of different characteristics of the image generated.

In some non-limiting embodiments or aspects, map generation system 102generates a plurality of images associated with one or morecharacteristics of the crosswalk based on the image data. As an example,map generation system 102 outputs an image representing each of theplurality of different characteristics. For example, map generationsystem 102 outputs one or more images including a classification of oneor more elements of the image, a segmentation of the one or moreelements of the image, and an angle of the one or more elements of theimage with respect to a line in the roadway.

In some non-limiting embodiments or aspects, map generation system 102outputs a classification of one or more elements of the image. Forexample, map generation system 102 outputs (e.g. generates, creates,etc.) an image including a classification of the one or more elements ofthe image that includes a distance map including a plurality of elementsassociated with at least one of the first boundary and the secondboundary, the distance map including at least one distance to the atleast one of the first boundary and the second boundary for at least oneelement of the plurality of elements within a threshold distance of theat least one of the first boundary and the second boundary. As anexample, map generation system 102 outputs a classification for anelement of the one or more elements associated with a threshold distancerelative to at least one of the first boundary and the second boundary.In some non-limiting embodiments or aspects, the distance map mayprovide an inverse distance transform from the at least one of the firstboundary and the second boundary that form an outline (e.g., a contour,etc.) around a crosswalk. In an example, the distance map may be limitedby a threshold distance (e.g., limited to a predetermined area, etc.)for a predetermined number of elements (e.g., 30 pixels, 1.2 meters,etc.)

In some non-limiting embodiments or aspects, map generation system 102outputs (e.g. generates, creates, etc.) a segmentation of one or moreelements of the image. As an example, map generation system 102 outputsan element by element foreground and background segmentation (e.g., apixel-wise segmentation, etc.) of a crosswalk. For example, a foregroundand background segmentation includes a segmentation of the one or moreelements of the image that includes a map of the crosswalk identifying aplurality of elements inside the one or more boundaries of thecrosswalk, and a plurality of elements outside the one or moreboundaries of the crosswalk. For example, the segmentation of the one ormore elements of the image includes a segmentation image including aplurality of elements inside the first boundary and the second boundaryand a plurality of elements outside the first boundary and secondboundary.

In some non-limiting embodiments or aspects, map generation system 102outputs (e.g. generates, creates, etc.) an angle of the one or moreelements of the image with respect to a line in the roadway. As anexample, map generation system 102 outputs an angle for an element ofthe one or more elements of the image. For example, map generationsystem 102 outputs an angle based on one or more angles associated witha position of the one or more elements with respect to the line of theroad. In some non-limiting embodiments or aspects, the line includes atleast a portion of a centerline of the roadway. In some non-limitingembodiments or aspects, map generation system 102 generates a pluralityof angles associated with a position of a plurality of elements. Forexample, map generation system 102 outputs a dilated representation of aboundary angle including a plurality of elements having an x value and ay value with respect to the line.

In some non-limiting embodiments or aspects, map generation system 102may use a SoftMax probability function to calculate a probabilitydistribution of the event over different events. The map generationsystem 102 determines the probabilities with a range (e.g. 0 to 1), andthe sum of all the probabilities will be equal to one. The mapgeneration system 102 uses the SoftMax function for multi-classificationmodel it returns the probabilities of each class and the target classwill have the high probability. In some cases, map generation system 102may output a vector of crosswalk classification.

In some non-limiting embodiments or aspects, map generation system 102trains the convolutional neural network by modifying parameters tominimize a loss over the loss function of the convolutional neuralnetwork that depends on the ground truth label and one or morepredictions, wherein the loss function comprises a sum of a segmentationloss, an alignment loss, and a boundary loss.

In some non-limiting embodiments or aspects, a boundary loss (e.g.,distance transform classification, etc.) and alignment loss (e.g., angleprediction, etc.) are modeled as regression tasks for predicting a valueand the segmentation loss as a pixel-wise classification task specifyinga category of a pixel of a crosswalk.

In some non-limiting embodiments or aspects, map generation system 102trains the convolutional neural network, by minimizing a sum of lossesover three prediction tasks according to the following Equation (1):

l(I)=lseg(I)+l _(dt)(I)+λ_(l) l _(a) (I)   (1)

In Equation (1), λ_(l) is a weighting variable for an alignment loss. Inan example, λ_(l)=100 as determined through cross-validation.

In some non-limiting embodiments or aspects, a segmentation loss lseg isdetermined to be binary cross entropy according to the followingEquation (2):

$\begin{matrix}{{{lseg}(I)} = {\frac{1}{N}{\sum_{p}( {{{\hat{y}}_{p}{\log ( y_{p} )}} + {( {1 - {\hat{y}}_{p}} ){\log ( {1 - y_{p}} )}}} )}}} & (2)\end{matrix}$

In Equation (2), N is the number of pixels in the bird's eye view image,ŷ_(p) represents the ground truth pixel p's value and y_(p) represents apredicted probability of p being a crosswalk.

In some non-limiting embodiments or aspects, a boundary loss l_(dt) isdetermined by a mean squared loss according to the following Equation(3):

$\begin{matrix}{{l_{dt}(I)} = {\frac{1}{N}{\sum_{p}{{d_{p} - {\hat{d}}_{p}}}^{2}}}} & (3)\end{matrix}$

In Equation (3), d_(p) is pixel p's value in the inverse distancetransform feature map ϕ_(dt).

In some non-limiting embodiments or aspects, alignment loss l_(a) isdefined as the mean squared loss according to the following Equation(4):

$\begin{matrix}{{l_{a}(I)} = {\frac{1}{N}{\sum_{p}{{{a\; {\tan ( \frac{v_{p,y}}{v_{p,x}} )}} - {\hat{\alpha}}_{p}}}^{2}}}} & (4)\end{matrix}$

In Equation (4), v_(p,y) and v_(p,x) are the y and x components of theunit vector corresponding to the predicted angle, and α_(p,gt) is theground truth angle. In an example, output is restricted to be between(0, π), as a single cross-walk boundary can be represented with multipleangles.

In some non-limiting embodiments or aspects, map generation system 102uses an objective function to train a model, that is either a lossfunction or its negative (e.g., a reward function, a profit function, autility function, a fitness function, etc.), in which case it is to bemaximized. In some non-limiting embodiments or aspects, map generationsystem 102 uses a combination of objective functions.

As shown in FIG. 4, at step 406, process 400 includes determining aposition of the crosswalk based on different characteristics. Forexample, map generation system 102 determines a position of thecrosswalk based on different characteristics. As an example, mapgeneration system 102 determines a position of the crosswalk based onthe plurality of different characteristics, wherein the positionincludes a first boundary and a second boundary of the crosswalk in theroadway.

In some non-limiting embodiments or aspects, map generation system 102receives the plurality of images associated with one or morecharacteristics of the crosswalk based on the image data from theconvolutional neural network. As an example, map generation system 102determines the position of the crosswalk based on receiving theplurality of images associated with one or more characteristics of thecrosswalk from the convolutional neural network.

In some non-limiting embodiments or aspects, process 400 includesdetermining a predicted boundary by maximizing a function. As anexample, for given output feature maps, ϕ_(seg) and ϕ_(dt), (e.g., ofthe segmentation and classification respectively, etc.), an energymaximization formulation, according to the following Equation (5):

$\begin{matrix}{{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}} & (5)\end{matrix}$

In Equation (5), map generation system 102 determines a first boundaryand a second boundary by providing the one or more candidate parametersin formulating an energy maximization problem, where potentials encodethe agreement between the segmentation and boundary semantic features.

In Equation (5), χ₁ and χ₂ are two predicted points on a centerline thatdefine a crosswalk, β is the boundary angle, λ_(I) is the weighting usedto balance between the segmentation and semantic edge feature maps, l isthe road center line. For example, ϕ_(seg) is a segmented output featuremap including the segmentation of the one or more elements in the image,ϕ_(dt) is a classification output feature map including theclassification of the one or more elements in the image, χ₁ and χ₂ arepoints on the line l in the roadway, β is the one or more angles of theone or more elements of the image with respect to the line l, and λ_(I)is a weight that balances ϕ_(seg) and ϕ_(dt).

In some non-limiting embodiments or aspects, an exhaustive search can becomputed very efficiently by using non-axis align integral accumulators(e.g., such as a gradient, etc.). In particular the ϕ_(seg) can beconverted to a 1D integral image along the road centerline which can beused to calculate the number of enclosed crosswalk pixels inside theboundaries defined by χ₁ and χ₂.

In some non-limiting embodiments or aspects, map generation system 102determines, based on the one or more angles and the image data, a firstlikelihood of the one or more elements being on the first boundary orthe second boundary, a second likelihood of the one or more elementsbeing between the first boundary and the second boundary, and based onthe first likelihood and the second likelihood, whether the position ofthe crosswalk on the roadway includes the one or more elements. In anexample, map generation system 102 determines, based on the one or moreangles and the image data, a first likelihood of the one or moreelements being on the first boundary or the second boundary. In anexample, map generation system 102 determines, based on the one or moreangles and the image data, a second likelihood of the one or moreelements being between the first boundary and the second boundary. In anexample, map generation system 102 determines, based on the firstlikelihood and the second likelihood, whether the position of thecrosswalk on the roadway includes the one or more elements.

In some non-limiting embodiments or aspects, boundary values of ϕ_(dt)(e.g., of inverse distance transform image, etc.) are at a maximum whenthey are associated with values on a boundary of a predicted crosswalkas opposed to near or adjacent, and favors a formulation with predictedboundaries that are right on a boundary in the inverse distancetransform image. Segmentation values of ϕ_(seg) form a segmentationpotential used to ensure the two boundaries have a maximum number ofcrosswalk pixels inside and a maximum number of background pixelsoutside.

In accordance with Equation (5), map generation system 102 determines aposition of the crosswalk based on a higher likelihood of a predictedcrosswalk based on a crosswalk having an element (e.g., an element of animage map associated with a crosswalk, a pixel in an image mapassociated with a crosswalk, etc.) in a boundary based on aclassification (e.g., an element of a first boundary, an element in asecond boundary, etc.), and/or includes a maximum number of crosswalkelements (e.g., a pixel, a region, etc.) inside a boundary segment and amaximum number of background elements outside a boundary segment of thesegmentation of the one or more elements of the image. For example, mapgeneration system 102 determines a position of a crosswalk by balancingϕ_(seg) and ϕ_(dt) in relation to an intersection polygon and acenterline (e.g., a mapped roadway, etc.) associated with the crosswalkuntil an optimal position (e.g., a best position, etc.) of theboundaries χ₁ and χ₂, and an angle are determined for the crosswalk. Insome non-limiting embodiments or aspects, map generation system 102determines an optimal angle for boundaries χ₁ and χ₂, by expanding theangle β for the one or more angles of the one or more elements of theimage to ensure an optimal number of elements of ϕ_(seg) and ϕ_(dt) areincluded in the crosswalk polygon. For example, map generation system102 may aggregate (e.g., determine an average value, a mode value, etc.)one or more angles taken along boundaries χ₁ and χ₂.

In some non-limiting embodiments or aspects, a map generation system 102determines a position of a crosswalk by calculating along a centerlineof a crosswalk position to maximize ϕ_(seg) and ϕ_(dt) to form alikelihood of the crosswalk until a maximum number of elements areenclosed in the crosswalk and a boundary includes a maximum number ofboundary elements of a crosswalk. In some non-limiting embodiments oraspects, map generation system 102 uses an integral accumulator todetermine an exact inference (e.g., nearly exact, etc.). For example,map generation system 102 determines one or more inferences based on aline in the road (e.g., driving path, road centerlines, etc.), anintersection polygon, and a plurality of different characteristics ofthe image based on the image data (e.g., feature maps, etc.) predictedby a multi-task convolutional network.

In some non-limiting embodiments or aspects, map generation system 102is programmed or configured to determine one or more inferences bysearching for optimal boundaries in the input data. As an example, mapgeneration system 102 is configured to perform a search for points on acenterline that maximize ϕ_(seg) and ϕ_(dt) before determining an anglethat further maximizes the formulation. For example, map generationsystem 102 determines the one or more inferences associated with acrosswalk to provide a prediction about where a crosswalk is mostaccurate with respect to ϕ_(seg) and ϕ_(dt)., before determining anangel.

In some non-limiting embodiments or aspects, map generation system 102selects a candidate angle (e.g., predicts an optimal angle, etc.) fromthe angle of the one or more elements of the roadway previouslygenerated. As an example, map generation system 102 continues tomaximize ϕ_(seg) and ϕ_(dt) based on a candidate angle. For example, mapgeneration system 102 determines additional angles for drawing acrosswalk based on a candidate angle by expanding the candidate angle(e.g., ±2°, ±5°, etc.) In some non-limiting embodiments or aspects, mapgeneration system 102 determines an angle of a boundary based anaggregation of angles along a centerline or a boundary, such as a modeof a prediction, an average, etc.

With continued reference to FIG. 4, at step 408, process 400 includesproviding map data associated with map of roadway including position ofcrosswalk. For example, map generation system 102 provides map dataassociated with map of roadway including position of crosswalk. As anexample, map generation system 102 provides map data associated with amap of the roadway, wherein the map data includes the position of thecrosswalk on the roadway in the map.

In some non-limiting embodiments or aspects, map generation system 102and/or autonomous vehicle 104 may determine a crosswalk in a roadwaybased on a predicted position of a crosswalk associated. In somenon-limiting embodiments or aspects, map generation system 102 and/orautonomous vehicle 104 may determine that an element of a crosswalk mapincludes a crosswalk and/or a crossing lane based on an automaticallygenerated crosswalk associated with an element of the crosswalk map.

Predicted Boundary Examples

In some non-limiting embodiments or aspects, a large dataset wascollected for a city, such that all crosswalks in the city within anarea of 100 km² were used. In total, 9,502 km were driven by autonomousvehicles and/or vehicles to create the dataset. As an example, thedataset included 1,571 training images, 411 validation images, and 607test images. As a further example, the dataset included 2,559intersections with 8,526 crosswalks. In such an example, the resultsinclude 5,203 training, 1,412 validation, and 1,911 test crosswalks andeach image represents an intersection with at least one crosswalk andhas a resolution of 4 cm per pixel.

In some non-limiting embodiments or aspects, precision and/or recallprovide based metrics. For example, precision provides a true positivewhich equals a set of predicted crosswalks with a minimal distancesmaller than τ and TP+FP=|P| and recall provides a true positive whichequals the set of ground truth crosswalks with minimal distance smallerthan τ and TP+FN=|G|. In an example, precision and recall are evaluatedat a τ of 20 cm, 40 cm, 60 cm, and 80 cm. In a further example, anIntersection over Union (“IOU”) may be calculated for the generatedcrosswalks and the ground truth.

In some non-limiting embodiments or aspects, map generation system 102trains a model using a batch size of 1. For example, training a model isperformed with a learning rate of 1e-4 and a weight decay of 5e-4 (e.g.,a model is optimized as described by Kingma, D. P., Ba, J. in “Adam: Amethod for stochastic optimization”, (In ICLR, 2015), the entirecontents of which is hereby incorporated by reference). In an example, alearning rate may decrease by a factor of 10 every 100,000 trainingiterations. In some non-limiting embodiments or aspects, map generationsystem 102 performs data augmentation when training by randomly flippingand rotating the images. In some non-limiting embodiments or aspects, amodels may be trained for 110 epochs over an entire training set.

Table 1 below provides performance metrics for implementations of amodel using various inputs. For example, in some implementations,different models are trained that use camera only, LIDAR only, acombination of both sensors (e.g., sensor modality), and/or the like. InTable 1, the columns N, C, and L identify number of passes (e.g., multi,single, etc.), camera input (e.g., RGB, etc.), and LIDAR input. Forexample, “Mutt” represents multiple passes over the map for offlinemapping and “1” represents a single pass over the map for onlinemapping. For example, online mapping is performed real-time as an AV orother mapping vehicle is traversing a roadway. As an example, an AV maydetect a crosswalk while traversing the crosswalk or traversing in closeproximity to the crosswalk. In such an example, an AV may process theimage as it passes through a geographic area (e.g., a city, anintersection, etc.) only once and map the crosswalk concurrently. Insome implementations, map generation system 102 performs offline mappingto draw a crosswalk. As an example, offline mapping is where one or moreAVs or vehicles traverse a roadway a plurality of times (e.g., manytimes) to acquire image data (e.g., a number of LIDAR and camera sweeps,etc.). For example, after collecting the image data, map generationsystem 102 aggregates the data (e.g., LIDAR and camera images) to mapthe crosswalks. In some non-limiting embodiments or aspects, offlinemapping may produce better results because a process can be implementedto ensure that every part of a roadway is present (e.g., missingareas/holes in the image data are accounted for, etc.). The firstbaseline in Table 1, (NN), provides a nearest neighbor algorithm withrespect to visual features. The second baseline in Table 1, (Seg),provides the segmentation output from the model trained on multiplepasses of a ground camera and LIDAR. In Table 1 results are compared fora single vehicle traversal (e.g., online mapping, etc.) to results usingseveral vehicle traversals to create an input feature map (e.g., offlinemapping, etc.). Table 1 shows the results of using multiple passes foroffline mapping perform better, with 96.6% (e.g., row 7, precision @ 40cm). Table 1 shows 91.5% automation can be reached (e.g., row 4,precision @ 40 cm) for online mapping. As further shown in Table 1,performance is better using a combination of both sensors in image data.

TABLE 1 Precision at (cm) Recall at (cm) N C L 20 40 60 80 20 40 60 80IOU NN Mult ✓ ✓ 21.4% 24.8% 25.2% 25.4% 19.4% 22.3% 22.7% 43.1% 35.9%Seg Mult ✓ ✓ 80.1% 93.1% 94.5% 95.0% 77.1% 91.9% 95.2% 97.1% 88.7% Ours1 ✓ — 78.8% 91.2% 93.8% 94.9% 78.6% 90.5% 92.9% 93.8% 86.9% Ours 1 — ✓77.2% 90.6% 93.1% 94.1% 76.8% 89.7% 91.9% 92.8% 85.7% Ours 1 ✓ ✓ 79.8%91.5% 93.6% 94.6% 79.9% 91.3% 93.2% 93.9% 87.1% Ours Mult ✓ — 83.4%94.9% 96.6% 97.3% 83.3% 94.6% 96.2% 96.8% 90.2% Ours Mult — ✓ 84.5%95.8% 97.6% 98.4% 85.0% 96.1% 97.8% 98.3% 91.8% Ours Mult ✓ ✓ 85.6%96.6% 98.1% 98.8% 86.1% 96.8% 98.2% 98.7% 92.4% Human — — — 88.3% 99.4%99.7% 99.8% 87.3% 98.3% 98.8% 98.8% 95.3%

Referring now to FIG. 7, FIG. 7 shows visualizations of implementationresults for a model trained on both camera and LIDAR in an offline mapsetting. For example, FIG. 7 shows visualizations of offline mappingresults performed when a model is trained on both camera and LIDARimagery, including comparisons between a) ground camera input, b) aground LIDAR input, c) predicted inverse distance transform, d)predicted segmentation, e) predicted crosswalk polygons after inference,and f) ground truth (“gt”) crosswalk polygons.

Referring now to FIG. 8, FIG. 8 shows visualizations of implementationresults for a model trained on an online map setting (e.g., drawingcrosswalks with very complex topology in both settings). For example,FIG. 8 shows online mapping results for a model trained on both cameraand LIDAR imagery. As shown in FIG. 8, comparisons between col a) groundcamera input (online map), b) ground LIDAR input (online map), c)predicted inverse distance transform, d) predicted segmentation, e)predicted crosswalk polygons after inference and f) ground truthcrosswalk polygons (e.g., ground truth overlaying a ground camera image(e.g., offline map), etc.).

Still referring to Table 1, the first entry in Table 1 shows results ofan implementation using a nearest neighbor algorithm on top of networkfeatures extracted from both a camera and LIDAR. For example, Table 1shows only a 24.8% automation (e.g., precision @ 40 cm) using a nearestneighbor (NN) algorithm as compared to a second entry in Table 1 (Seg)showing results of using only the output of a semantic segmentationimage generated by the convolutional neural network (CNN) model for afinal prediction. As shown in Table 1, the network performs at up to93.1% automation (e.g., precision @ 40 cm) in the offline setting usinga semantic segmentation only. In an example, Table 1 shows humandisagreement in drawing crosswalks (e.g., noise in human annotations,etc.) by comparing implementations of ground truth including annotationsof 100 intersections generated with several annotators (e.g., mapproducers, etc.). As shown in the last row in Table 1, there is about a4.7% error in IOU, an 11.7% error in the precision, and a 12.7% error inrecall at 20 cm between different individuals.

In some non-limiting embodiments or aspects, a CNN provides a forwardpass measured at 50 microseconds per image. For example, a determinationof a first boundary and a second boundary may be performed in 0.75 s ona single core CPU. FIG. 7 and FIG. 8 are examples showing qualitativeresults of implementations in a complex topology to accurately determinea crosswalk.

Referring now to FIG. 9, FIG. 9 provides a histogram of Intersectionover Union (“IOU”) results for a model using both LIDAR and cameraimages as input. For example, FIG. 9 shows a histogram of the IOUs(left) and a cumulative IOU graph (right) using LIDAR and camera imagesas input. FIG. 9 shows that 94.1% of the crosswalk images have an IOUgreater than 85.0%.

In implementations, determining a correct crosswalk angle may providehigh performance cross walk generation (e.g., drawing, etc.). Forexample, to determine an accuracy of crosswalk angles, an analysis of acombination of a predicted alignment and centerline angle is compared toa ground truth over the same crosswalk to find a difference between anangle used in inference with a ground truth angle. Referring now to FIG.10, FIG. 10 shows a histogram of angle differences (left) and cumulativeangle differences (right) using an offline model trained on both cameraand LIDAR. For instance, FIG. 10 shows an example of plotting ahistogram and cumulative graph of the differences. In an example, ananalysis is provided for a model trained on both the camera and LIDARimagery from the offline maps. As an example, FIG. 10 provides metricsbased on analysis showing that 89% of the crosswalk angles measured arewithin ±5 ° from a ground truth. In some implementations, this becomesmuch higher (e.g., 98%) after a structured prediction step (e.g. afterexpanding a search to include additional angles, etc.).

In some non-limiting embodiments or aspects, intersection complexity isanalyzed to show an effect of a number of neighboring roads on modelresults. As an example, a neighboring road is defined as one of one ormore roads connecting to an intersection that provides a road centerlinefor a structured prediction. As an example, if a roadway has a dividerin a middle, the roadway is split into 2 roads. In some examples, a 4way intersection may have 8 roads, such as, for example, 2 roads foreach approach to an intersection. Referring now to FIGS. 11A-11C, FIGS.11A-11C show for implementations, as a number of roads increases (e.g.,for intersections that are more complex, etc.), performance decreases.As shown in FIGS. 11A-11C, a visualization of an effect on a number ofneighboring roads is provided based on results of a model trained onoffline maps using both camera and LIDAR. For example, FIG. 11A providesa histogram of Intersection over Union (“IOU”) compared to a number ofneighboring roads, FIG. 11B provides precision at 40 cm compared to anumber of neighboring roads, and FIG. 11C provides recall at 40 cm. ascompared to a number of neighboring roads.

Table 2 below provides results of an ablation study for implementationsto analyze the contributions of different components in our model in thecontext of offline mapping with cameras and LIDAR. Table 2 shows animplementation for ablation and performance using oracle information.For example, an ablation study can be performed to analyze an effect ofan angle search, road centerline angles, and predicted angles in rows.In an implementation, for oracle information ground truth distancetransform, segmentation and angles are provided to analyze results inrows 5-8. For example, Table 2 shows an effect of removing certaincomponents of the model. For example, in implementations of Table 2,results are shown when an angle search of ±2 ° and ±5 ° is removed inrow 2 and the usage of the centerline angle is removed in row 3. Forexample, Table 2 shows a result of a slight decrease in performance. InTable 2, row 4 a predicted angle is not used when drawing crosswalks,showing a significant drop of more than 10% for all the performancemetrics.

In implementations, oracle information is used to analyze an upper boundperformance of a model. In Table 2, by comparing rows 5, 6 and 7, it isshown that having ground truth segmentation significantly increases theperformance of the model. Alternatively, Table 2 shows having groundtruth distance transforms only slightly increases the performance. In anexample, Table 2 shows that performance using a ground truth angle wasonly slightly worse than the result in row 1 due to the accuracy of thepredicted angles. Table 2 shows that without ground truth angles themodel can already achieve 98% angle accuracy. As a further example,Table 2 shows examples of having an alignment prediction correlating toconsiderably good inference results.

TABLE 2 Precision at (cm) Recall at (cm) 20 40 60 80 20 40 60 80 IOUOurs 85.6% 96.6% 98.1% 98.8% 86.1% 96.8% 98.2% 98.7% 92.4% No Ang Search82.2% 94.4% 97.1% 98.1% 82.7% 94.7% 97.2% 98.2% 91.3% No Cent Ang 84.5%96.3% 98.1% 98.8% 84.9% 96.4% 98.0% 98.6% 92.1% No Pred Ang 74.0% 85.3%88.9% 91.4% 73.8% 84.8% 88.3% 90.5% 83.7% GT DT 88.5% 96.6% 97.8% 98.3%89.5% 97.3% 98.4% 98.8% 92.9% GT Seg 94.1% 97.8% 98.7% 99.2% 94.7% 98.1%98.8% 99.1% 94.9% GT Ang 85.5% 96.5% 98.1% 98.7% 85.7% 96.4% 97.9% 98.4%92.2% GT DT + S + A 93.9% 97.5% 98.5% 99.0% 94.9% 98.1% 98.9% 99.2%94.9%

In some non-limiting embodiments or aspects, map generation system 102uses a weight λ_(I) to weigh between maximizing segmentation or distancetransform energies in an energy formulation. For example, referring nowto FIG. 12, FIG. 12 shows a main failure mode caused by balancingcharacteristics between a segmentation and distance transform weights(top) and poor image/paint quality (bottom). As shown in FIG. 12,comparisons are provided between a) ground camera input, b) ground LIDARinput, c) predicted inverse distance transform, d) predictedsegmentation, e) predicted crosswalk polygons after inference, and f)ground truth crosswalk polygons.

As shown in FIG. 12 in the top image, since almost half of the crosswalkboundary in the right crosswalk is missing, an implementation of themodel predicts the wrong segmentation. In this case, the implementationof the model shows that predicting a boundary focusing on a segmentationenergy provides a larger value and may produce an incorrect inference.In a further example shown in FIG. 12, a second failure mode is shown ina bottom image, having a poor paint quality in ground imagery (althoughnot shown, this is also true for the LIDAR imagery). Thus, in someimplementations a model may mistake a crosswalk (e.g., a boundary line,etc.) for a stop line at an intersection, and may not predict itspresence for the segmentation output.

Referring now to FIG. 13, FIG. 13 shows failure modes of an onlinemapping implementation. As an example, FIG. 13 shows comparisons betweena) ground camera input (e.g., online map), b) ground LIDAR input (e.g.,online map), c) predicted inverse distance transform, d) predictedsegmentation, e) predicted crosswalk polygons after inference and f)ground truth crosswalk polygons overlaid on the ground camera (e.g.,offline map). For example, failure modes are shown as holes in the map.In an example, a main failure mode for an online model is caused by poordata collection when mapping a roadway that may be associated with holesand/or poor image quality as shown in FIG. 13.

In some non-limiting embodiments or aspects, an implementation of amodel trained with only crosswalks (e.g., not trained with images ofintersections without crosswalks presented, etc.) produces 5.7% falsepositives. In some non-limiting embodiments or aspects, a modelretrained with images that do not contain crosswalks (e.g., 45% addedimages of intersections without crosswalks, etc.), a false positive rateis 0.04%, similar to a performance result from Table 1 row 8. Forexample, referring now to FIG. 14, FIG. 14 shows examples of results fora model retrained with images where a crosswalk is missing. As anexample, FIG. 14 shows comparisons between a) ground camera input, b)ground LIDAR input, c) predicted inverse distance transform, d)predicted segmentation, e) predicted crosswalk polygons, f) ground truthpolygons.

In some non-limiting embodiments or aspects, map generation system 102may generate stop lines if only a first boundary is predicted instead oftwo boundaries. In some non-limiting embodiments or aspects, mapgeneration system 102 may determine a road/intersection boundaryprediction. For example, referring now to FIG. 15, FIG. 15 shows animplementation of a model where a method can be applied to determine aroad/intersection boundary prediction. In such an implementation, amodel outputs both an inverse distance transform and predictedsegmentation, and uses a driving path, all mover priors, and/or the likeat one or more intervals to perform a search perpendicular to thevehicle path for a left and right point of a boundary. For example, aleft and right point of a boundary can be further extended to draw alane (e.g., a road edge, a drivable surface boundary, etc.).

Although embodiments or aspects have been described in detail for thepurpose of illustration and description, it is to be understood thatsuch detail is solely for that purpose and that embodiments or aspectsare not limited to the disclosed embodiments or aspects, but, on thecontrary, are intended to cover modifications and equivalentarrangements that are within the spirit and scope of the appendedclaims. For example, it is to be understood that the present disclosurecontemplates that, to the extent possible, one or more features of anyembodiment or aspect can be combined with one or more features of anyother embodiment or aspect. In fact, many of these features can becombined in ways not specifically recited in the claims and/or disclosedin the specification. Although each dependent claim listed below maydirectly depend on only one claim, the disclosure of possibleimplementations includes each dependent claim in combination with everyother claim in the claim set.

What is claimed is:
 1. A computer-implemented method, comprising:receiving, with a computing system comprising one or more processors,image data associated with an image of a roadway including a crosswalk;generating, with the computing system, a plurality of differentcharacteristics of the image based on the image data, wherein theplurality of different characteristics include a classification of oneor more elements of the image, a segmentation of the one or moreelements of the image, and one or more angles of the one or moreelements of the image with respect to a line in the roadway;determining, with the computing system, a position of the crosswalk onthe roadway based on the plurality of different characteristics, whereinthe position includes a first boundary and a second boundary of thecrosswalk in the roadway; and providing, with the computing system, mapdata associated with a map of the roadway, wherein the map data includesthe position of the crosswalk on the roadway in the map.
 2. Thecomputer-implemented method of claim 1, wherein determining the positionof the crosswalk on the roadway based on the plurality of differentcharacteristics further comprises: determining, based on the one or moreangles and the image data, a first likelihood of the one or moreelements being on the first boundary or the second boundary;determining, based on the one or more angles and the image data, asecond likelihood of the one or more elements being between the firstboundary and the second boundary; and determining, based on the firstlikelihood and the second likelihood, whether the position of thecrosswalk on the roadway includes the one or more elements.
 3. Thecomputer-implemented method of claim 1, wherein the position of thecrosswalk on the roadway is determined according to the followingfunction:${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁ and χ₂ are points on the line l inthe roadway, β is the one or more angles of the one or more elements ofthe image with respect to the line l, and λ₁ is a weight that balancesϕ_(seg) and ϕ_(dt).
 4. The computer-implemented method of claim 1,wherein the segmentation of the one or more elements of the imageincludes a segmentation image including a plurality of elements insidethe first boundary and the second boundary and a plurality of elementsoutside the first boundary and second boundary.
 5. Thecomputer-implemented method of claim 1, wherein the classification ofthe one or more elements of the image includes a distance map includinga plurality of elements associated with at least one of the firstboundary and the second boundary, the distance map including at leastone distance to the at least one of the first boundary and the secondboundary for at least one element of the plurality of elements within athreshold distance of the at least one of the first boundary and thesecond boundary.
 6. The computer-implemented method of claim 1, whereinthe line includes at least a portion of a centerline of the roadway. 7.The computer-implemented method of claim 1, wherein the one or moreangles of the one or more elements comprise a dilated representation ofa boundary angle including an x-value and an y-value with respect to theline.
 8. The computer-implemented method of claim 1, further comprising:training, with the computing system, a convolutional neural network bymodifying parameters to optimize a loss function of the convolutionalneural network that depends on ground truth labels and one or morepredictions of the plurality of different characteristics, wherein theloss function comprises a sum of a segmentation loss, an alignment loss,and a boundary loss; and processing, with the computing system, theimage data associated with the image of the roadway including thecrosswalk using the trained convolutional neural network, to generate aplurality of images associated with the plurality of differentcharacteristics of the image.
 9. A computing system, comprising: one ormore processors programmed and/or configured to: receive image dataassociated with an image of a roadway including a crosswalk; generate aplurality of different characteristics of the image based on the imagedata, wherein the plurality of different characteristics include aclassification of one or more elements of the image, a segmentation ofthe one or more elements of the image, and one or more angles of the oneor more elements of the image with respect to a line in the roadway;determine a position of the crosswalk on the roadway based on theplurality of different characteristics, wherein the position includes afirst boundary and a second boundary of the crosswalk in the roadway;and provide map data associated with a map of the roadway, wherein themap data includes the position of the crosswalk on the roadway in themap.
 10. The computing system of claim 9, wherein the one or moreprocessors are further programmed or configured to determine theposition of the crosswalk on the roadway based on the plurality ofdifferent characteristics by: determining, based on the one or moreangles and the image data, a first likelihood of the one or moreelements being on the first boundary or the second boundary;determining, based on the one or more angles and the image data, asecond likelihood of the one or more elements being between the firstboundary and the second boundary; and determining, based on the firstlikelihood and the second likelihood, whether the position of thecrosswalk on the roadway includes the one or more elements.
 11. Thecomputing system of claim 9, wherein the position of the crosswalk onthe roadway is determined according to the following function:${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁ and χ₂ are points on the line l inthe roadway, β is the one or more angles of the one or more elements ofthe image with respect to the line l, and λ₁ is a weight that balancesϕ_(seg) and ϕ_(dt).
 12. The computing system of claim 9, wherein thesegmentation of the one or more elements of the image includes asegmentation image including a plurality of elements inside the firstboundary and the second boundary and a plurality of elements outside thefirst boundary and second boundary.
 13. The computing system of claim 9,wherein the classification of the one or more elements of the imageincludes a distance map including a plurality of elements associatedwith at least one of the first boundary and the second boundary, thedistance map including at least one distance to the at least one of thefirst boundary and the second boundary for at least one element of theplurality of elements within a threshold distance of the at least one ofthe first boundary and the second boundary.
 14. The computing system ofclaim 9, wherein the line includes at least a portion of a centerline ofthe roadway.
 15. The computing system of claim 9, wherein the one ormore angles of the one or more elements comprise a dilatedrepresentation of a boundary angle including an x value and a y valuewith respect to the line.
 16. The computing system of claim 9, whereinthe one or more processors are further programmed and/or configured to:train a convolutional neural network by modifying parameters to optimizea loss function of the convolutional neural network that depends onground truth labels and one or more predictions of the plurality ofdifferent characteristics, wherein the loss function comprises a sum ofa segmentation loss, an alignment loss, and a boundary loss; and processthe image data associated with the image of the roadway including thecrosswalk using the trained convolutional neural network, to generate aplurality of images associated with the plurality of differentcharacteristics of the image.
 17. A computer program product comprisingat least one non-transitory computer-readable medium including programinstructions that, when executed by at least one processor, cause the atleast one processor to: receive image data associated with an image of aroadway including a crosswalk; generate a plurality of differentcharacteristics of the image based on the image data, wherein theplurality of different characteristics include a classification of oneor more elements of the image, a segmentation of the one or moreelements of the image, and one or more angles of the one or moreelements of the image with respect to a line in the roadway; determine aposition of the crosswalk on the roadway based on the plurality ofdifferent characteristics, wherein the position includes a firstboundary and a second boundary of the crosswalk in the roadway; andprovide map data associated with a map of the roadway, wherein the mapdata includes the position of the crosswalk on the roadway in the map.18. The computer program product of claim 17, wherein the instructionscause the at least one processor to determine the position of thecrosswalk on the roadway based on the plurality of differentcharacteristics by: determining, based on the one or more angles and theimage data, a first likelihood of the one or more elements being on thefirst boundary or the second boundary; determining, based on the one ormore angles and the image data, a second likelihood of the one or moreelements being between the first boundary and the second boundary; anddetermining, based on the first likelihood and the second likelihood,whether the position of the crosswalk on the roadway includes the one ormore elements
 19. The computer program product of claim 17, wherein theposition of the crosswalk on the roadway is determined according to thefollowing function:${{\max\limits_{\chi_{1},\chi_{2},\beta}{\lambda_{I}( {\varphi_{{seg},,\beta^{(\chi_{2})}} - \varphi_{{seg},,\beta^{(\chi_{1})}}} )}} + {( {1 - \lambda_{I}} )( {\varphi_{{dt},,\beta^{(\chi_{2})}} + \varphi_{{dt},,\beta^{(\chi_{1})}}} )}},$wherein ϕ_(seg) is a segmented output feature map including thesegmentation of the one or more elements in the image, ϕ_(dt) is aclassification output feature map including the classification of theone or more elements in the image, χ₁ and χ₂ are points on the line l inthe roadway, β is the one or more angles of the one or more elements ofthe image with respect to the line l, and λ_(I) is a weight thatbalances ϕ_(seg) and ϕ_(dt).
 20. The computer program product of claim17, wherein the line includes at least a portion of a centerline of theroadway.